What Is Hallucination: A Plain-English Definition
Hallucination in LLMs occurs when the model generates text that is confidently stated but factually incorrect, unsupported by the provided context, or entirely invented. It is a direct consequence of how LLMs are trained. The model learns to predict the next token based on statistical patterns in its training data. When asked about something outside its training distribution, or when asked to recall specific facts that are not strongly represented in its training data, the model may generate a plausible-sounding response based on partial patterns rather than correct information. Hallucinations take several forms. Factual hallucinations are the most dangerous: the model states an incorrect fact confidently, such as citing a court case that does not exist, attributing a quote to the wrong person, or stating incorrect statistics. Contextual hallucinations occur when the model contradicts information explicitly provided in the prompt, ignoring evidence in favour of its trained patterns. Logical hallucinations involve reasoning errors where the model arrives at an incorrect conclusion from correct premises. And fabrication hallucinations involve the model inventing entire entities, whether companies, people, studies, or publications, that do not exist. The severity of hallucinations varies by model and task. Factual recall tasks where the answer depends on specific memorised information are highest risk. Summarisation tasks where the source document is provided are lower risk, though models can still add or modify facts. Open-ended generation tasks carry moderate risk. Understanding which parts of your product are hallucination-prone and designing appropriately for each is essential.
How Hallucination Happens
LLMs are trained by repeatedly predicting the next token in a vast corpus of text. This produces a model with excellent understanding of language structure, facts that appear frequently in training data, and the ability to generate fluent, confident prose. But fluency and confidence are independent of accuracy. A model can generate a very fluent, very confident, very wrong sentence. Several factors increase hallucination risk. Recency is one: models have training cutoffs and will not know about events that occurred after their training data was collected. Specificity is another: asking for a specific number, date, citation, or name is riskier than asking for a general explanation, because specific facts require accurate retrieval while general explanations can be assembled from broad patterns. Adversarial framing increases risk: questions phrased in ways that assume a false premise often lead the model to confirm the premise rather than correct it. Consider a concrete example of the problem. A UK law firm builds a legal research assistant and asks it to 'find cases about restrictive covenants in employment contracts.' The model, having been trained on law review articles and case summaries, can convincingly describe case holdings with specific case names and citations. Some of those citations are real. Some are fabricated but plausible-sounding. A junior solicitor who does not verify every citation before use could cite a non-existent case, causing significant professional and legal risk. This is why hallucination mitigation is not optional in professional AI products.
Why Hallucination Matters for AI Product Development
Hallucination matters because it is the most visible failure mode of AI products and the one most likely to destroy user trust. One high-profile hallucination incident where a user acts on incorrect AI-generated information can cause significant harm, generate negative press coverage, and result in regulatory scrutiny. In regulated sectors, the stakes are particularly high. Financial AI products where the model invents market data or misrepresents regulations are a material risk under FCA oversight. Health AI products that hallucinate clinical information face MHRA scrutiny and potential liability. Legal AI tools that fabricate case citations have already resulted in court sanctions for the legal professionals who relied on them without verification. Even in lower-stakes domains, hallucination erodes the trust that drives AI product adoption. An AI tool that users have to constantly fact-check provides less value than a more reliable tool. Reducing hallucination is directly correlated with user trust, retention, and willingness to pay. The compliance angle under the EU AI Act and UK GDPR is also relevant. AI Act transparency obligations require high-risk AI systems to document their limitations, which includes acknowledging hallucination risks. Products that make claims about AI reliability without disclosing hallucination risks may face regulatory challenge.
Common Use Cases Where Hallucination Is Reduced
The most effective architectural response to hallucination is retrieval-augmented generation. By providing the model with the specific, authoritative documents it needs to answer a question, you shift the task from knowledge recall (where hallucination is common) to knowledge synthesis from provided text (where hallucination is significantly reduced). A model grounded in a provided document is far less likely to invent facts than one asked to recall from memory. Structured output constraints reduce a specific type of hallucination. If the model must return a JSON object with defined fields and a schema validator rejects invalid responses, the model cannot freely hallucinate content in unconstrained fields. Combine this with instructions to return 'null' or 'unknown' for fields where information is not available rather than guessing. Citation-required prompting is an effective technique for factual tasks. Instructing the model to cite the specific document passage supporting each claim, then verifying those citations programmatically, creates an audit trail that catches fabricated citations before they reach users. Confidence flagging is another pattern: instructing the model to rate its confidence in each claim and highlighting low-confidence content to the user so they know what to verify. This does not reduce hallucination but manages its impact. For NHS Digital applications or MHRA-regulated medical software, additional safeguards are required beyond these technical measures. Human-in-the-loop review of AI-generated content, clear labelling of AI-generated outputs, and escalation pathways for uncertain information are baseline requirements for clinical AI applications in the UK.
Related Concepts You Need to Know
Retrieval-augmented generation is the most important mitigation for hallucination in production AI products. By providing the model with retrieved, authoritative context before asking it to generate a response, you dramatically reduce reliance on memorised knowledge and the associated hallucination risk. AI guardrails are safety layers applied to model outputs before they reach users. Guardrails can include output validation against a schema, fact-checking against a knowledge base, moderation classifiers that flag potentially incorrect claims, and human review queues for high-stakes outputs. Prompt engineering techniques directly affect hallucination rates. Instructing the model to say 'I don't know' when uncertain, requiring it to cite sources, and using chain-of-thought prompting to encourage explicit reasoning all reduce hallucination on specific task types. Structured output constrains the model's generation space, reducing the scope for creative fabrication. Combining structured output requirements with explicit 'unknown' field values rather than guessing creates more reliable, auditable AI features. AI transparency is the regulatory and ethical obligation to disclose how AI-generated content is produced and what its limitations are. For products where hallucination risk is non-trivial, transparency means clearly labelling AI-generated content and helping users understand what they should verify independently.