Hallucination: when AI invents convincing falsehoods
Definition
A hallucination is an answer from a language model that sounds fluent and plausible but is factually wrong or entirely made up. The model predicts the most likely sequence of words, not verified truth. Without a link to solid sources it can produce names, numbers, paragraphs or citations that never existed, all while sounding fully confident.
Why the Term Matters
Whoever deploys AI is liable for the output, not the model. An invented legal statement, a wrong price or a non-existent product variant can flow straight into contracts, quotes or customer messages. As a neutral AI consultancy we frame this risk early so an initiative does not fail on a pretty demo but stays reliable in daily use. The term separates feasibility from accountability.
When It Is Relevant
- A chatbot or assistant gives answers that customers rely on.
- Answers flow unchecked into documents, quotes or support tickets.
- Factual accuracy is mandatory, for example in law, engineering or finance.
- Leadership wants to know before rollout how errors are caught.
When Not
For pure creative tasks like idea collection, draft copy or brainstorming, an occasional invention rarely hurts because a human selects anyway. For internal summaries with the source right next to them, heavy safeguarding is often overkill. What matters is whether anyone reuses the output unchecked.
Industries That Use It
- Tax and legal advice: a misquoted statute is a liability, so every statement needs a verifiable source.
- Machinery and plant engineering: invented technical values in quotes or maintenance notes cause costly errors.
- Insurance: terms and coverage sums must be exact, approximation is not enough here.
- E-commerce: wrong product details in customer chat create returns and complaints.
Typical Use Cases
- A support assistant that answers only from approved manuals.
- A quoting helper that pulls prices only from the released system.
- An internal knowledge assistant that backs every statement with source and location.
- A check step that routes unsourced answers to human review automatically.
- A test series with typical trick questions to measure the error rate before launch.
Decision Guide
| Signal | Safeguard needed | Alternative |
|---|---|---|
| Customer relies on the answer | retrieval with mandatory sources | free generation is too risky |
| Internal ideation only | light review suffices | a human selects anyway |
| Law, finance, engineering involved | strict sourcing and sign-off | unchecked rollout is dangerous |
| No factual basis, pure wording | model may write freely | sourcing would be overkill |
Example
A vendor wants to launch a customer assistant for product questions. During the potential analysis, test prompts show the model invents values when information is missing. We recommend binding the assistant to a verified product catalogue and showing a source with every answer. Questions with no match go to a human. This describes the method, not an asserted client metric.
Service Connection
The risk becomes visible early in the AI potential analysis, clean delivery with source binding is part of AI implementation, and legal duties are framed by EU AI Act consulting.
Sources
This framing describes common practice in working with language models. Statements on liability and legal duties are general orientation and not legal advice; they must be checked case by case against the currently applicable legal texts.