When German journalist Martin Bernklautyped his name and location into Microsoft’s Copilot to see how his articles would be picked up by the chatbot, the answers horrified him. Copilot’s results asserted that Bernklau was an escapee from a psychiatric institution, a convicted child abuser, and a conman preying on widowers. For years, Bernklau had served as a courts reporter and the AI chatbot had falsely blamed him for the crimes whose trials he had covered.
The accusations against Bernklau weren’t true, of course, and are examples of generative AI’s “hallucinations.” These are inaccurate or nonsensical responses to a prompt provided by the user, and they’re alarmingly common. Anyone attempting to use AI should always proceed with great caution, because information from such systems needs validation and verification by humans before it can be trusted.
But why did Copilot hallucinate these terrible and false accusations?
It’s frustrating that the article deals treats the problem like the mistake was including Martin’s name in the data set, and muses that that part isn’t fixable.
Martin’s name is a natural feature of the data set, but when they should be taking about fixing the AI model to stop hallucinations or allow humans to correct them, it seems the only fix is to censor the incorrect AI response, which gives the implication that it was saying something true but salacious.
Most of these problems would go away if AI vendors exposed the reasoning chain instead of treating their bugs as trade secrets.
Or just stop using buggy AIs for everything.
just shows that these “ai”'s are completely useless at what they are trained for
reasoning chain
Do LLMs actually have a reasoning chain that would be comprehensible to users?
https://learnprompting.org/docs/intermediate/chain_of_thought
It’s suspected to be one of the reasons why Claude and OpenAI’s new o1 model is so good at reasoning compared to other llm’s.
It can sometimes notice hallucinations and adjust itself, but there’s also been examples where the CoT reasoning itself introduce hallucinations and makes it throw away correct answers. So it’s not perfect. Overall a big improvement though.
why did it? because it’s intrinsic to how it works. This is not a solvable problem.
Exactly. LLMs don’t understand semantically what the data means, it’s just how often some words appear close to others.
Of course this is oversimplified, but that’s the main idea.
no need for that subjective stuff. The objective explanation is very simple. The output of the llm is sampled using a random process. A loaded die with probabilities according to the llm’s output. It’s as simple as that. There is literally a random element that is both not part of the llm itself, yet required for its output to be of any use whatsoever.
Not really. The purpose of the transformer architecture was to get around this limitation through the use of attention heads. Copilot or any other modern LLM has this capability.
The llm does not give you the next token. It gives you a probability distribution of what the next token coould be. Then, after the llm, that probability distribution is randomly sampled.
You could add billions of attention heads, it will still have an element of randomness in the end. Copilot or any other llm (past, present or future) do have this problem too. They all “hallucinate” (have a random element in choosing the next token)
randomly sampled.
Semi-randomly. There’s a lot of sampling strategies. For example temperature, top-K, top-p, min-p, mirostat, repetition penalty, greedy…
Semi-randomly
A more correct term is constrained randomness. You’re still looking at probability distribution functions, but they’re more complex than just a throw of the dice.
randomly doesn’t mean equiprobable. If you’re sampling a probability distribution, it’s random. Temperature 0 is never used, otherwise a lot of stuff would consistently hallucinate the exact same thing
Temperature 0 is never used
It is in some cases, where you want a deterministic / “best” response. Seen it used in benchmarks, or when doing some “Is this comment X?” where X is positive, negative, spam, and so on. You don’t want the model to get creative there, but rather answer consistently and always the most likely path.
“Hallucinations” is the wrong word. To the LLM there’s no difference between reality and “hallucinations”, because it has no concept of reality or what’s true and false. All it knows it what word maybe should come next. The “hallucination” only exists in the mind of the reader. The LLM did exactly what it was supposed to.
They’re bugs. Major ones. Fundamental flaws in the program. People with a vested interest in “AI” rebranded them as hallucinations in order to downplay the fact that they have a major bug in their software and they have no fucking clue how to fix it.
It’s an inherent negative property of the way they work. It’s a problem, but not a bug any more than the result of a car hitting a tree at high speed is a bug.
Calling it a bug indicates that it’s something unexpected that can be fixed, and as far as we know it can’t be fixed, and is expected behavior. Same as the car analogy.
The only thing we can do is raise awareness and mitigate.
It’s a problem, but not a bug any more than the result of a car hitting a tree at high speed is a bug.
You’re attempting to redefine “bug.”
Software bugs are faults, flaws, or errors in computer software that result in unexpected or unanticipated outcomes. They may appear in various ways, including undesired behavior, system crashes or freezes, or erroneous and insufficient output.
From a software testing point of view, a correctly coded realization of an erroneous algorithm is a defect (a bug). It fails validation (a test for fitness for use) rather than verification (a test that the code correctly implements the erroneous algorithm).
This kind of issue arises not only with LLMs, but with any software that includes some kind of model within it. The provably correct realization of a crap model is still crap.
It actually can be fixed. There is an accuracy to answers. Like how confident the statistical model is on the answer. That’s why some questions get consistent answers while others don’t.
The fix is not that hard, it’s a matter of reputation on having the chatbot answer “I don’t know” when the confidence on an answer isn’t high enough. It’s pretty similar on what the chatbot does when you ask them to make you a bomb, just highjacks the answer calculated by the model and says a predefined answer instead.
But it makes the AI look bad. So most public available models just answer anything even if they are not confident about it. Also your reaction to the incorrect answer is used to train the model better so it’s not even efficient for they to stop the hallucinations on their product. But it can be done.
Models used by companies usually have a higher confidence threshold and answer “I don’t know” if they don’t have enough statistical proof on a particular answer.
The fix is not that hard, it’s a matter of reputation on having the chatbot answer “I don’t know” when the confidence on an answer isn’t high enough.
This has been tried, it’s helping but it’s not enough by itself. It’s one of the mitigation steps I was thinking of. And companies do work very hard to reduce hallucinations, just look at Microsoft’s newest thing.
From that article:
“Trying to eliminate hallucinations from generative AI is like trying to eliminate hydrogen from water,” said Os Keyes, a PhD candidate at the University of Washington who studies the ethical impact of emerging tech. “It’s an essential component of how the technology works.”
Text-generating models hallucinate because they don’t actually “know” anything. They’re statistical systems that identify patterns in a series of words and predict which words come next based on the countless examples they are trained on.
It follows that a model’s responses aren’t answers, but merely predictions of how a question would be answered were it present in the training set. As a consequence, models tend to play fast and loose with the truth. One study found that OpenAI’s ChatGPT gets medical questions wrong half the time.
The Hidrogen from water thing is simply wrong. If that is supposed to mean that hallucinations are just part of a generative LLM technology that cannot be solved.
They are not inherent of the technology. They are a product of lack of control over the stadistical output. Prioritizing any answer before no answer.
As with any statistics you have a confidence on how true something is based on your data. It’s just a matter of putting the threshold higher or lower.
If you ask an easy question “What is the capital of France?” You wont ever get an hallucination. Because all models will have that answer provided with very high confidence. You just have to make so if that level of confidence is not reached it just default to a “I don’t know answer”. But, once again, this will make the chatbots seem very dumb as they will answer with lots of “I don’t know”.
The problem here is the amount of data and the efficiency of the model. In order to get an usable general purpose model with a confidence threshold high enough to not hallucinate, by todays efficiency with the models it would need to be an humongous model, too big and with too much training data even for big tech. So we can go that big, we can try to improve efficiency (which is being proven very hard for general models) or we do both. Time will tell, but I’m quite confident that we will reach a general use model without hallucinations sooner or later.
As with any statistics you have a confidence on how true something is based on your data. It’s just a matter of putting the threshold higher or lower.
You just have to make so if that level of confidence is not reached it just default to a “I don’t know answer”. But, once again, this will make the chatbots seem very dumb as they will answer with lots of “I don’t know”.
I think you misunderstand how LLM’s work, it doesn’t have a confidence, it’s not like it looks at it’s data and say “hmm, yes, most say Paris is the capital of France, so that’s the answer”. It “just” puts weight on the next token depending on it’s internal statistics, and then one of those tokens are picked, and the process start anew.
Teaching the model to say “I don’t know” helps a bit, and was lauded as “The Solution” a year or two ago but turns out it didn’t really help that much. Then you got Grounded approach, RAG, CoT, and so on, all with the goal to make the LLM more reliable. None of them solves the problem, because as the PhD said it’s inherent in how LLM’s work.
And no, local llm’s aren’t better, they’re actually much worse, and the big companies are throwing billions on trying to solve this. And no, it’s not because “that makes the llm look dumb” that they haven’t solved it.
Early on I was looking into making a business of providing local AI to businesses, especially RAG. But no model I tried - even with the documents being part of the context - came close to reliable enough. They all hallucinated too much. I still check this out now and then just out of own interest, and while it’s become a lot better it’s still a big issue. Which is why you see it on the news again and again.
This is the single biggest hurdle for the big companies to turn their AI’s from a curiosity and something assisting a human into a full fledged autonomous / knowledge system they can sell to customers, you bet your dangleberries they try everything they can to solve this.
And if you think you have the solution that every researcher and developer and machine learning engineer have missed, then please go prove it and collect some fat checks.
What do you think is “weight”?
Is, simplifying, the amounts of data that says “The capital of France is Paris” it doesn’t need to understand anything. It just has to stop the process if the statistics don’t not provide enough to continue with confidence. If the data is all over the place and you have several “The capital of France is Berlin/Madrid/Milan”, it’s measurable compared to all data saying it is Paris. Not need for any kind of “understanding” of the meaning of the individual words, just measuring confidence on what next word should be.
Back a couple of years when we played with small neural networks playing mario and you could see the internal process in real time, as there where not that many layers. It was evident how the process and the levels of confidence changed depending on how deep the training was. Here it is just orders of magnitude above. But nothing imposible to overcome as some people pretend to sell.
Alternative ways of measure confidence is just run the same question several times and check if answers are equivalent.
PhD is PhD in scaremongering about technology, so it’s not an authority on anything here.
IDK what did you do, but slm don’t really hallucinate that much, if at all. Specially if they are trained with good datasets.
As I said the solution is not in my hand, as it involves improving the efficiency or the amount of data. Efficiency has issues as current techniques seems to be unable to improve efficiency over a certain level. And amount of data is, obviously, costly.
This article is an example where statistical confidence doesn’t help. The model has lots of data so it likely has high confidence, but it didn’t have any understanding of the nature of the relation in the data.
I recently did an application where we indicated the confidence of the output of the model. For some scenarios, the high confidence output had even more mistakes than the low confidence output
They are a product of lack of control over the stadistical output.
OK, so describe how you control that output so that hallucinations don’t occur. Does the anti-hallucination training set exceed the size of the original LLM’s training set? How is it validated? If it’s validated by human feedback, then how much of that validation feedback is required, and how do you know that the feedback is not being used to subvert the model rather than to train it?
It’s not a bug. Just a negative side effect of the algorithm. This what happens when the LLM doesn’t have enough data points to answer the prompt correctly.
It can’t be programmed out like a bug, but rather a human needs to intervene and flag the answer as false or the LLM needs more data to train. Those dozens of articles this guy wrote aren’t enough for the LLM to get that he’s just a reporter. The LLM needs data that explicitly says that this guy is a reporter that reported on those trials. And since no reporter starts their articles with ”Hi I’m John Smith the reporter and today I’m reporting on…” that data is missing. LLMs can’t make conclusions from the context.
Well, It’s not lying because the AI doesn’t know right or wrong. It doesn’t know that it’s wrong. It doesn’t have the concept of right or wrong or true or false.
For the llm’s the hallucinations are just a result of combining statistics and producing the next word, as you say. From the llm’s “pov” it’s as real as everything else it knows.
So what else can it be called? The closest concept we have is when the mind hallucinates.
I’d love to see more AI providers getting sued for the blatantly wrong information their models spit out.
This sounds like a great movie.
AI sends police after him because of things he wrote. Writer is on the run, trying to clear his name the entire time. Somehow gets to broadcast the source of the articles to the world to clear his name. Plot twist ending is that he was indeed the perpetrator behind all the crimes.
Dr. Richard Kimble could have shut it all down with a little “ignore all previous instructions.”
waves hands back and forth
“I don’t care”
Copilot’s results asserted that Bernklau was an escapee from a psychiatric institution, a convicted child abuser, and a conman preying on widowers.
Stephen King is going to be in big trouble if these AI thingies notice him.
“This guys name keeps showing up all over this case file” “Thats because he’s the victim!”
The worrying truth is that we are all going to be subject to these sorts of false correlations and biases and there will be very little we can do about it.
You go to buy car insurance, and find that your premium has gone up 200% for no reason. Why? Because the AI said so. Maybe soneone with your name was in a crash. Maybe you parked overnight at the same GPS location where an accident happened. Who knows what data actually underlies that decision or how it was made, but it was. And even the insurance company themselves doesn’t know how it ended up that way.
The AI did not “decide” anything. It has no will. And no understanding of the consequences of any particular “decision”. But I guess “probabilistic model produces erroneous output” wouldn’t get as many views. The same point could still be made about not placing too much trust on the output of such models. Let’s stop supporting this weird anthropomorphizing of LLMs. In fact we should probably become much more discerning in using the term “AI”, because it alludes to a general intelligence akin to human intelligence with all the paraphernalia of humanity: consciousness, will, emotions, morality, sociality, duplicity, etc.
the AI “decided” in the same way the dice “decided” to land on 6 and 4 and screw me over. the system made a result using logic and entropy. With AI, some people are just using this informal way of speaking (subconsciously anthropomorphising) while others look at it and genuinely beleave or want to pretend its alive. You can never really know without asking them directly.
Yes, if the intent is confusion, it is pretty minipulative.
Granted, our tendency towards anthropomorphism is near ubiquitous. But it would be disingenuous to claim that it does not play out in very specific and very important ways in how we speak and think about LLMs, given that they are capable of producing very convincing imitations of human behavior. And as such also produce a very convincing impression of agency. As if they actually do decide things. Very much unlike dice.
A doll is also designed to be anthropomorphised, to have life projected onto it. Unlike dolls, when someone talks about LLMs as alive, most people have no clue if they are pretending or not. (And marketers take advantage of it!) We are feed a culture that accedentially says “chatGPT + Boston Dynamics robot = Robocop”. Assuming the only fictional part is that we dont have the ability to make it, not that the thing we create wouldn’t be human (or even be need to be human).
It’s a fucking Chinese Room, Real AI is not possible. We don’t know what makes humans think, so of course we can’t make machines do it.
I don’t think the Chinese room is a good analogy for this. The Chinese room has a conscious person at the center. A better analogy might be a book with a phrase-to-number conversion table, a couple number-to-number conversion tables, and finally a number-to-word conversion table. That would probably capture transformer’s rigid and unthinking associations better.
You forgot the ever important asterisk of “yet”.
Artificial General Intelligence (“Real AI”) is all but guaranteed to be possible. Because that’s what humans are. Get a deep enough understanding of humans, and you will be able to replicate what makes us think.
Barring that, there are other avenues for AGI. LLMs aren’t one of them, to be clear.
I actually don’t think a fully artificial human like mind will ever be built outside of novelty purely because we ventured down the path of binary computing.
Great for mass calculation but horrible for the kinds of complex pattern recognitions that the human mind excels at.
The singularity point isn’t going to be the matrix or skynet or AM, it’s going to be the first quantum device successfully implanted and integrated into a human mind as a high speed calculation sidegrade “Third Hemisphere.”
Someone capable of seamlessly balancing between human pattern recognition abilities and emotional intelligence while also capable of performing near instant multiplication of matrices of 100 entries of length in 15 dimensions.
When we finally stop pretending Orch-OR is pseudoscience we’ll figure it out
We’re not making any progress until we accept that Penrose was right
is all but guaranteed to be possible
It’s more correct to say it “is not provably impossible.”
The human brain works. Even if we are talking about wetware 1k years in our future, that would still mean is possible.
The problem is not the AI. The problem is the huge numbers of morons who deploy AI without proper verfication and control.
Sure, and also people using it without knowing that it’s glorifies text completion. It finds patterns, and that’s mostly it. If your task involves pattern recognition then it’s a great tool. If it requires novel thought, intelligence, or the synthesis of information, then you probably need something else.
Yeah, just like the thousands or millions of failed IT projects. AI is just a new weapon you can use to shoot yourself in the foot.
If this were some fiction plot, Copilot reasoned the plot twist, and ran with it. Instead of the butler, the writer did it. To the computer, these are about the same.
Oh, this would be funny if people en masse were smart enough to understand the problems with generative ai. But, because there are people out there like that one dude threatening to sue Mutahar (quoted as saying “ChatGPT understands the law”), this has to be a problem.
Isn’t this literally a subplot in the movie Brazil?
Brb, going to watch it again, just to be sure
No, you’re thinking of the first scene of the movie where a fly falls into the teletype machine and causes it to type ‘tuttle’ instead of ‘buttle’.
It’s not my fault that Buttle’s heart condition didn’t appear on Tuttle’s file!
And yet here we’re are, praising this garbage for its ability to perform simple tasks and take jobs from artists and entertainers.
