ELI5: What does it mean when a large language model (such as ChatGPT) is "hallucinating," and what causes it?

Traditionally, Hallucination is a type of error for tasks that have a "ground truth", and rather than leaving something out, it adds something in. For example, if a model is tasked with summarising a text, and it adds something that wasn't in the original.

For LLMs, this term is not entirely appropriate, except in the context of the task YOU have decided it should do. It's just a language model, generating text that sounds reasonable, so in that sense it's always doing what it's supposed to be doing, and not hallucinating in the traditional sense. But it's also reasonable to use this as a description of an error for a tasks you define. For example, if you type "Who is Noam Chomsky" and it generates the text 'Noam Chomsky is a linguist who wrote "The Perks of Being a Wallflower"', you can argue that hallucination is the right characterisation of the error IF your task is to get it to generate text that you can interpret as true facts about Noam Chomsky.

It's a bit vague, more of a term for hand-done error analysis than a clearly defined term. For example, Noam Chomsky wrote The Minimalist Program, but if it said he wrote The Minimal Programs, is that a hallucination or different kind of error?

There's a classic technique in computer science & programming circles called a Markov generator where a work of text, like Moby Dick, is analyzed for how often any word appears after every other word. Like, for all the times the word "the" is in the book, the next word is "whale" 10% of the time, "harpoon" 5%, "ocean" 3%, etc... Then you run the process in reverse - pick a word, then pick the next word randomly, but with a probability according to how often it appeared after the first word, and then pick a third word the same way, etc, forever.

It's clear that there is nothing in this process that knows the words' meaning, or the sentences, or the subject they're about. It just knows what words are used alongside each other. LLMs are a lot like this, except on a larger scale. They take more words into account when they're choosing their next word, and the way it comes up with the probabilities of the next words is more complicated, but no less mechanical.

If you’d like to see what hallucinations look like, triggering an obvious one would help you understand better than explanations of why they happen. 

Take your favorite TV show and ask it for a plot summary of one of the most popular episodes, something likely to be in its training data because it’s been reviewed and talked about all over the internet. It’ll give you a great summary and review. 

Now ask it about one of the lesser known filler episodes. You’ll probably get a plot summary of a completely fictional episode that sounds like it could have been an episode of that show, but it wasn’t, and certainly wasn’t the episode you asked about. That’s a hallucination. 

It’s not necessarily emotional answers. It gives answers that sound right. That’s it, nothing more than that. There’s no fact checking, or trying to be correct. It typically sounds correct because the vast majority of writing prior to AI is correct because the author cared about accuracy.

So if you ask it to write something that might not exist, it may fill in that blank with something that sounds right…but isn’t. For example, if you want it to write a legal briefing, those typically use references to existing supporting cases or legal situations. So if those supporting cases don’t actually exist, then the AI will “hallucinate” and make references that sound right (but don’t actually exist).

Large Language Models (LLMs) dont really "know" anything, but are in essence extremely advanced predictive texting programs. They work in a fundamentally different way to older chatbot and predictive text programs, but the outcome is the same: they generate text that is likely to come next, without any coherent understanding of what it's talking about.

Thus, when asked about something factual, it will created a response that is statistically likely to be correct, based on its training data. If its well trained, theirs a decent chance it will generate the "correct" answer simply because that is the likely answer to that question, but it doesn't have a concept of the question and the facts being asked of it, just a complex "black box" series of relationships between various tags in its training data and what is a likely response is to that input.

Sometimes, when asked that factual question, it comes up with an answer that statistically likely, but just plain WRONG, or just make it up as it goes. For example, thier was an AI generated legal filing that just created citations to non-existent cases to support its case.

This is what they are talking about when they say its "hallucinating", which is a almost deliberately misleading term, becuase it implies the AI can "think", whereas it never "thinks" as we understand thoughts, just consults a enormous lookup table and returns a series of outputs.

You might have heard that machine learning models are made of many neurons.

A model that has a single neuron is called a perceptron. So think you have a neuron and you are trying to train it to identify whether John will go to the circus given some situation.

Your neuron will have 1 output (likelihood of John going to the circus) and 3 inputs (whether Sarah is going, whether Marcus is going, whether Anna is going).

You will feed this model with many past experiences and adjust the function to accommodate them. Eventually the model will learn that it’s a safe bet to say you are going to the circus EVERY TIME Sarah is there, you ALWAYS avoid being alone with Anna, and you go to the circus if Marcus and someone else is there

Great, but the real world is more complex than that. There are situations beyond those. Maybe it’s raining, maybe you ate something bad, maybe you’ve seen Sarah everyday for the past month and wants a break from her. Point is, even if the model gives the best results on average given its training data, it might still make mistakes.

Real machine learning models have lots of neurons, all connected in different and complex organizations. Their “thinking” and their “mistakes” all inform the next one in complex ways

This article provides a good explanation and is written by an expert:

Peter J. Denning. 2025. In Large Language Models We Trust? Commun. ACM 68, 6 (June 2025), 23–25. https://doi.org/10.1145/3726009

LLM is essentially an autocomplete.

Recall those dumb old memes about autocomplete saying really dumb things to complete the sentence. The words are in order, but the sentences don't make sense.

Now, apply it to a larger scale like paragraphs and essays. That's "hallucinating"

Hallucination is just an euphemism for it being wrong.

Everything an LLM tells you is made up. It's just that sometimes it makes stuff up that is actually true. It has no way of knowing whether what it says is true or not though.

Some people like to pretend that it hallucinates sometimes when this happens, when in fact it's, to the llm, just business as usual.

Nothing to do with "emotional" answers.

They are large statistical engines. They aren't capable of original thought. They just take their training data and regurgitate parts of it according to the statistics of how relevant they appear to be to the question. With large amounts of training data, they are able to regurgitate something for most things you ask of it.

However, when their training data doesn't cover what they're being asked, they don't know how to respond. They're just dumb statistical machines. The stats don't add up for any part of their data, in that instance, so they tend to go a bit potty when asked something outside the bounds of their training. Thus it appears as though they've gone potty, and start "imagining" (FYI they are not imagining anything) things that don't exist.

So if the statistics for a question seem to succeed 96% of the time when they give the answer "cat", they'll answer "cat" for those kinds of questions, or anything similar.

But when they're asked a question they just don't have the training data for, or anything similar, there is no one answer that looks correct 96% of the time. Or even 50% of the time. Or at all. So what happens is they can't select the surest answer. It just isn't sure enough. There is no answer in their training data that was accepted as a valid answer 96% of the time for that kind of question. So they are forced to dial down and find words that featured, say, 2% as valid answers to similar questions.

This means that, effectively, they start returning any old random nonsense because it's no more nonsense than any other answer. Or it's very, very slightly LESS nonsensical to talk about pigeons according to their stats when they have no idea of the actual answer.

And so they insert nonsense into their answers. Or they "hallucinate".

The LLM does not know how to say "I don't know the answer". That was never programmed into its training data as "the correct response" because it just doesn't have enough data to cater for the situation it's found itself in... not even the answer "I don't know". It was never able to form a statistical correlation between the question asked (and all the keywords in it) and the answer "I don't know" for which it was told "That's the correct answer".

The "training" portion of building an LLM costs billions and takes years and it is basically throwing every bit of text possible at it and then "rewarding" it by saying "Yes, that's a good answer" when it randomly gets the right answer. This is then recorded in the LLM training as "well... we were slightly more likely to get an answer that our creator tells us was correct when cats and baby were mentioned if our answer contained the word kitten". And building those statistical correlations, that's the process we call training the LLM.

When those statistical correlations don't exist (e.g. if you never train it on any data that mentions the names of the planets), it simply doesn't find any strong statistical correlation in its training data for the keywords "name", "planet", "solar system", etc. So what it does is return some vague and random association that is 0.0000001% more likely to have been "rewarded" during training for similar keywords. So it tells you nonsense like the 3rd planet is called Triangle. Because there's a vague statistical correlation in its database for "third", "triangle" and "name" and absolutely nothing about planets whatsoever. That's an hallucination.

All of the other responses here are giving reductionist "LLMs are just text predictors" answers, so I will try to give a more technical and nuanced explanation.

As seen in Anthropic's Tracing the Thoughts of a Large Language Model research, LLMs have an internal predictor of whether they know the answer to the question. Think of it like a neuron in a human brain: if the LLM knows the answer then the neuron turns on, and if it doesn't know the answer then it stays off. Whether that neuron fires determines whether the LLM gives an answer or says it doesn't know.

For example, when the LLM is asked "What did Micheal Jordan do?", the neuron will initially be off. As each layer of the LLM's neural network is computed, the model checks for stored information about Micheal Jordan. Once it finds a set of neurons corresponding to "Micheal Jordan played basketball", the "I know the answer" neuron will fire and the LLM will say "basketball". If it doesn't find a fact like that, then the "I know the answer" neuron will stay turned off, and the LLM will say "I don't know".

Anthropic's research found that hallucinations (the model giving the wrong answer) are often caused by faulty activation of this neuron. Basically, the model thinks it knows the answer when it doesn't. This is sometimes caused by (sticking with the above example) the neural network having a "Micheal Jordan" neuron that fires but not actually having the "played basketball" portion. When that happens it causes the network to spit out whatever information was stored where "played basketball" should have been, usually leading to an incorrect answer like "tennis".

This is a simplification of how these things work, and I abuse the word "neuron" to make it more understandable. I encourage people to read Anthropic's work to get a better understanding.

Also note that we didn't specifically build an "I know the answer" neuron into the model, it just spontaneously appeared through the wonders of deep learning.

Whoever says that is wrong. AI models don't have scripts and they certainly don't have emotions. "Hallucination" is just the term for when an AI model generates false, misleading, or nonsensical information.

LLMs aren't "thinking" like we do - they have no actual self-awareness about the responses they give. For the most part, all they do is figure out what the next word should be based on all the words that came before. Behind the scenes, the LLM is using all sorts of weighted connections between words (and maybe phrases) that enable it to determine what the next word/phrase it should use is, and once it's figured that out, what the next word/phrase it should use is, etc. There's no ability to determine truth or "correctness" - just the next word, and the next and the next.

If the LLM has lots and lots of well-developed connections in the data its been trained on, it will constantly reinforce those connections. And if those connections arise from accurate/true data, then for the most part, the connections will produce accurate/true answers. But if the connections arise (at least in part) from inaccurate/false data, then the words selected can easily lead to misleading/false responses. But there's no ability for th LLM to understand that - it doesn't know whether the series of words it selected to write "New York City is the capital of New York State" is accurate or true (or even what a city or state or capital is). If the strongest connections it sees in its data produce that sentence, then it will produce that sentence.

Similarly, if it's prompted to provide a response to something where there are no strong connections, then it will use weaker (but still relatively strong) connections to produce a series of words. The words will read like a well informed response - syntactically and stylistically the response will be no different from a completely accurate response - but will be incorrect. Stated with authority, well written and correct sounding, but still incorrect. These incorrect statements are hallucinations.

The Chinese Room Thought Experiment is crucial to understanding what LLM's are.

Imagine a man who only reads & speaks English, is locked in a room where he can speak to no one.

His only outside contact is a man who only speaks & reads Chinese.

Neither is aware of the identity of the other. Neither can talk to the other.

The man outside writes questions in Chinese on paper and slides them under the door. The man inside doesn't know what's written, but thankfully, has a huge book. Any question written in Chinese is in the book, along with an answer. He just has to find the page where the symbols match the paper.

Once he matches the symbols on the paper to the book, he copies the answer onto the paper and slides it back under the door.

The person outside believes he is conversing with a fluent Chinese speaker.

However, the man inside knows neither what the question says or what the answer says, he's just matching the symbols to their answer and giving that answer.

This is what LLM's are.

In their case, they're finding answers to each part of your question, and writing a set of symbols that most closely match each one put together.

It can't "know" if a detail it writes is wrong or not.

It's just the man in the Chinese room.