Why AI Makes Things Up

If you’ve used AI even a little, you’ve probably seen examples where the response is just silly. It makes something up, contradicts itself, or wanders off topic - all while sounding completely confident and plausible. This behavior is commonly called a hallucination. AI hallucinations occur when an AI system produces information that sounds convincing but is false, misleading, or unsupported.

Here’s an example using ChatGPT 4¹:

Note that the AI continues to insist that 3821 is not prime, inventing factors to prove its case. Even correcting individual mistakes, it still clings to the original assertion.

Another example comes from the New York Times. On April 14, 2026 they published an article that included a quote from Pierre Poilievre. The problem is, he never made that statement. The quote was generated by AI and was not caught by either the author or the editor before publication. (Two weeks later, the Times added an editor’s note correcting the error.)

What are Hallucinations?

To understand what is going on here, you need to understand how large language models (LLMs) work: they generate likely next tokens based on patterns they learned during training. They are completion engines, not truth machines. When an AI hallucinates, it is simply doing what it was designed to do.

If the model has incomplete information, conflicting patterns in training data, or a prompt that nudges it toward speculation, it may generate an answer that sounds coherent but is actually invented.

In the prime number example above, the response sounded plausible:

“No, 3821 is not a prime number. It can be divided evenly by 3, 29, 41, 87, 123, and 127, in addition to 1 and itself.”

The answer suggests that the model performed the factorization of 3821 and demonstrated that it therefore cannot be prime. But the LLM isn’t performing mathematical reasoning. It’s predicting a sequence of tokens that statistically resembles a correct answer.

The same phenomenon likely explains the fabricated quote in The New York Times. The model has learned the format of quotations and political speech. It knows what a quote looks like so it generates one. Given enough context about a public figure, it can generate a quote that sounds like something that person might say - even if they never actually said it.

Why Hallucinations Are Hard to Detect

What makes hallucinations hard to spot is the added tone of confidence that LLMs imbue. They don’t hedge or provide any hint of uncertainty. The model didn’t say

“I don’t believe 3821 is prime, but I’d need to verify with a small program.”

LLMs state uncertain things with strong confidence. This is partly because human-written training text is usually assertive, and “confident-sounding prose” statistically resembles good answers.

Hallucinations become even more likely when the model lacks relevant training data. That can happen because of training cut-off dates, because the information is proprietary or private, or the topic is highly specialized and underrepresented in training. This is one reason why context matters so much. Modern AI systems often perform dramatically better when they are supplied with relevant documents, search results, or authoritative data sources.

How AI Systems Are Improving

If you try the prime number example today with a current version of Chat GPT or Claude, you will probably get the correct answer. These applications are getting better at avoiding hallucinations through a number of strategies.

AI applications can search the web for relevant content and provide that information as additional context to the LLM. This helps ground the answer in the retrieved information. The applications can also use additional tools to run code, perform calculations, and query authoritative APIs to provide more accurate information instead of “guessing.”

These improvements involve the use of tools in the surrounding application environment. There have also been improvements within the LLM itself. After training, the models are tuned to be more likely to admit uncertainty, cite sources, and defer when confidence is low.

Guarding Against Hallucinations

Humans are actually quite poor at detecting hallucinations in areas where they lack expertise. A convincing hallucination often sounds fluent, uses correct terminology, follows logical structure, and “feels” authoritative. That makes hallucinations dangerous precisely because they often don’t look wrong.

One of the best techniques to detect hallucinations is to challenge the AI’s response. Ask it for the source that supports a claim or data point. Ask if the information is verified or inferred. Ask the LLM if there are parts of its response it is uncertain about. These types of follow-up questions often expose weaknesses in the answer or force the model to “fess up” to its errors, like with the prime number example.

Hallucinations are not really “bugs” in the traditional sense. They are a consequence of how probabilistm language models work. AI companies continue improveming systems to reduce their occurence, but it will likely be difficult - perhaps impossible - to eliminate them entirely.

AI can be incredibly useful and sometimes astonishingly capable. But for now, it remains a technology best used with curiousity, verification, and a healthy dose of skepticism