Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI. learn more
Every month, over 500 million people trust Gemini and ChatGPT to stay informed about everything from pasta to sex to homework. But if an AI tells you to cook pasta in gasoline, you shouldn’t accept advice about birth control or algebra either.
At the World Economic Forum in January, OpenAI CEO Sam Altman offered stark reassurance: But please explain your reasoning and decide if it is reasonable to me. …I think our AI systems will be able to do the same thing. They can walk you through the steps from A to B so you can decide if it’s a good step or not. ”
knowledge requires justification
No wonder Altman wants us to believe that a large-scale language model (LLM) like ChatGPT can generate transparent explanations for everything it says. Without good reason, what humans believe or suspect to be true can never amount to knowledge. Why not? Now, think about when you can confidently say that you know something. Perhaps it is when you feel absolutely confident in your beliefs because they are well supported by evidence, arguments, and the testimony of trusted authorities.
LLM aims to be a trusted institution. trusted information provider. But unless they can explain why, we cannot know whether their claims meet the standard of justification. For example, suppose you say today’s fog in Tennessee is caused by wildfires in western Canada. I might just take your word for it. But suppose you really swore to me yesterday that snake fights are a routine part of dissertation defense. Then you know you can’t be completely trusted. So you might ask, why do we think smog is caused by Canadian wildfires? For my beliefs to be valid, it is important to know that your reports are reliable.
The problem is that today’s AI systems cannot be trusted by sharing the reasoning behind what they say. Because no such reason exists. LLM is not designed for remote reasoning. Instead, models are trained on vast amounts of human writing to detect, predict, or enhance complex patterns in language. When a user enters a text prompt, the response is just the algorithm’s prediction of how the pattern is most likely to continue. These outputs (increasingly) convincingly mimic what a knowledgeable human would say. But the underlying process has nothing to do with whether the output is legitimate, much less true. As Hicks, Humphreys, and Slater state in “ChatGPT is Bullshit,” LLM is “designed to produce text that appears to be true without actually caring about the truth.” It is.
So if AI-generated content is not the artificial equivalent of human knowledge, what is it? Hicks, Humphries, and Slater are right to call it bullshit. Still, much of what the LLM spews is true. When these “bullshit” machines produce factually accurate output, they produce what philosophers call Gettier cases (after the philosopher Edmund Gettier). These incidents are interesting because they combine true belief in a strange way with ignorance about the validity of that belief.
AI output can be a mirage
Consider this example from the writings of the 8th century Indian Buddhist philosopher Dharmottala. Imagine we are asking for water on a hot day. We suddenly see water or think so. In fact, what we are seeing is not water but a mirage, but when we arrived at the scene, we were lucky enough to find water under a rock. Can we say that we have true knowledge about water?
People widely agree that whatever knowledge there is, the traveler in this example does not have it. Instead, they were lucky to find water exactly where they never believed they would find it.
The problem is, every time we think we know what we learned from LLM, we end up in the same shoes as the Dharmottala travellers. If the LLM is trained on a high-quality dataset, its assertion will likely be true. Those claims can be likened to a mirage. And just as water gushing out from under a rock turns out to be real, there’s probably evidence and arguments somewhere in that data set that can justify that claim. However, the legitimate evidence and arguments that probably existed played no role in the output of the LLM. Just as the presence of water played no role in creating an illusion that confirmed the traveler’s belief that there would be water there.
Mr. Altman’s reassurance is therefore highly misleading. If you ask an LLM to justify its output, what will it do? That’s not a real justification. This provides Gettier’s justification. A natural language pattern that convincingly imitates justification. Chimera of justice. As Hicks and others say, it’s a bullshit justification. As everyone knows, that’s not justified at all.
Currently, AI systems regularly get confused or have “hallucinations” like a mask that keeps falling off. But as the illusion of justification becomes more convincing, one of two things will happen.
For those who understand that true AI content is the big deal at Gettier, LLM’s patently false claim that it explains its own reasoning undermines its credibility. We see that AI is deliberately designed and trained to systematically deceive.
And those of us who don’t realize that the AI spits out Gettier rationalizations, false ones? Well, we’re just fooled. Realizing that as long as we rely on LLM, we will be living in a kind of pseudo-matrix, unable to distinguish between fact and fiction and needing to worry that there may be a difference. It will disappear.
Each output must be aligned
When weighing the significance of this predicament, it is important to keep in mind that there is nothing wrong with LLM functioning as usual. These are incredibly powerful tools. And people who understand that AI systems spit out Gettier cases instead of (artificial) knowledge are already using LLM in ways that take that into account. Programmers use LLM to draft code and then use their own coding expertise to modify the code according to their own standards and objectives. Professors use LLM to create essay prompts and revise them according to their own educational objectives. During this election cycle, any speechwriter worth his or her name will be checking a candidate’s AI-generated manuscript before letting him walk on stage. and so on.
But most people turn to AI precisely where expertise is lacking. Think of teenagers studying algebra or prophylaxis. Or some seniors are looking for advice on diet or investments. If an LLM is to mediate public access to such types of sensitive information, it needs to know, at a minimum, whether and when the LLM can be trusted. And gaining trust requires knowing what LLM doesn’t tell you: whether and how each output is justified.
Luckily, you probably know that olive oil is much better than gasoline when it comes to cooking spaghetti. But what is a dangerous recipe that has swallowed reality whole without ever tasting its legitimacy?
Hunter Kallay is a doctoral student in philosophy at the University of Tennessee.
Dr. Christina Gehrman is an associate professor of philosophy at the University of Tennessee.
data decision maker
Welcome to the VentureBeat community!
DataDecisionMakers is a place where experts, including technologists who work with data, can share data-related insights and innovations.
If you want to read about cutting-edge ideas, updates, best practices, and the future of data and data technology, join DataDecisionMakers.
Why not consider contributing your own articles?
Read more about DataDecisionMakers
