Mitigate LLM Hallucinations Step by Step

Why a hallucination can’t be «fixed» with a prompt

Before touching configuration, it’s worth accepting an uncomfortable reality: hallucinations are not a bug that gets eliminated with a firmer instruction. They are an emergent behavior of the model’s probabilistic functioning. An LLM generates the most probable token given a sequence; it doesn’t consult a database of facts. When the context doesn’t contain the answer, the model doesn’t stop: it completes with whatever statistically fits. That means any configuration that promises «zero hallucinations» is selling you smoke.

What you can do is reduce the surface area where they appear, detect them before they reach the user, and degrade the system in a controlled way when confidence drops. This article is a real configuration sequence, with concrete parameters, thresholds, and verifications, for teams that already have an LLM in production or are about to deploy one.

Step 1: separate the types of hallucination before configuring anything

Not all hallucinations are mitigated the same way. Configuring without classifying leads to patching the wrong place. Distinguish at least these four, because each one requires a different layer:

• Factual: the model asserts a false datum (a date, a figure, a law). It’s countered with context retrieval and external verification.
• Attribution: it invents a source, an author, or a quote. It’s the most dangerous in editorial content because it appears verifiable.
• Instruction: it ignores a system constraint (responds in another language, breaks the format). It’s countered with output validation.
• Reasoning: the intermediate steps are plausible but the conclusion doesn’t follow. It’s countered with decomposition and checking each step.

A case I see repeatedly in content teams: they set temperature to 0 thinking that puts an end to invented sources. It lowers variability, yes, but a model at temperature 0 still invents references if it doesn’t have the data. Temperature reduces dispersion, not ignorance.

Step 2: set temperature and sampling according to the task

Sampling configuration is the first real dial. There is no universal value; there is a value per task type. Follow this table:

Expert warning: lowering top_p to 0.5 trims the tail of improbable tokens, which reduces rare inventions but also makes the text flatter and more repetitive. In structured extraction tasks it’s worth combining it with constrained decoding (grammars or forced JSON schemas), which is more effective than any temperature adjustment for avoiding invented fields.

Step 3: set up retrieval with a rejection threshold

The most powerful lever against factual hallucination is not letting the model answer from memory. Configure a RAG (retrieval-augmented generation) pipeline with these explicit decisions:

1. Chunking: fragments of 300 to 600 tokens with 10-15% overlap. Fragments that are too large dilute the signal; too small and they lose the context that prevents erroneous interpretations.
2. Hybrid retrieval: combine semantic search (embeddings) with lexical search (BM25). Semantics fails with identifiers, codes, and proper nouns; lexical fails with paraphrases. Together they cover more.
3. Similarity threshold: define a minimum score (for example, 0.75 in cosine similarity, adjusted after measuring with your own corpus) below which the system doesn’t call the model and returns a «I don’t have enough information» message.
4. Closed-context prompt: instruct the model to answer only with the retrieved fragments and to explicitly indicate when the context doesn’t cover the question. This instruction isn’t infallible, but combined with the threshold it drastically reduces invented answers.

Here’s the practical mistake almost everyone makes the first time: setting the retrieval threshold too high. The system starts rejecting legitimate questions and the team, frustrated, lowers it to zero. The correct threshold isn’t guessed: it’s calibrated with a set of 50 to 100 real questions, measuring how many times it rejects a question that did have an answer versus how many times it responds without context. Without that calibration set, any threshold is a hunch.

Step 4: system instructions that actually work (and those that don’t)

Phrases like «don’t make anything up» have limited effectiveness. Operational and verifiable instructions work better:

• Instead of «be precise,» use «cite the context fragment each statement comes from; if there’s no fragment, respond ‘not on record.'»
• Instead of «don’t hallucinate,» use «if the question requires data later than your knowledge or not present in the context, say so before answering.»
• Add an output format with mandatory fields: answer, sources (list of fragments used), confidence (high/medium/low). The sources field forces the model to commit and facilitates automatic verification.

What doesn’t work: stacking ten negative constraints in the system prompt. Beyond a certain number, the model starts ignoring the first ones or over-restricting itself and rejecting valid requests. Prioritize three or four critical rules and validate each one with tests.

Step 5: output verification before showing the user

Configuration doesn’t end at the model. Add a programmatic validation layer between the response and the user:

• Schema validation: if you expect JSON, reject any output that doesn’t comply with the schema. Don’t try to repair it with another prompt without flagging the failure.
• Entity checking: extract names, dates, and figures from the response and compare them with the retrieved context. If an entity doesn’t appear in any source, flag it.
• Citation verification: if the model cites a document, check that the document exists in your index and that the cited fragment is in it. Invented citations are detected this way with high reliability.
• Second pass with another model or prompt: a smaller model acting as a verifier can flag unsupported statements. It isn’t infallible and adds latency, but in editorial content workflows it pays off.

Important nuance: automatic verification has false positives. A verifier that’s too strict will reject correct answers that paraphrase the context instead of quoting it literally. Configure the verifier to compare meaning, not exact string matching, or you’ll have a system that rejects everything.

Step 6: define the degradation behavior

A mature system knows what to do when it isn’t sure. Explicitly configure these states:

1. High confidence: direct answer with sources.
2. Medium confidence: answer with a visible warning and links to the sources for human verification.
3. Low confidence: don’t answer; offer to rephrase the question or escalate to a human.
4. Validation failure: log the complete case (prompt, retrieved context, output, reason for rejection) for later analysis.

Logging is the most forgotten part and the most valuable. Without a failure log you can’t improve thresholds or prompts; you can only pray. According to the documentation of the main model providers, continuous evaluation with your own test sets is the only reliable way to measure whether your changes reduce hallucinations or just shift them.

Step 7: measure with your own test set, not with public benchmarks

Generic benchmarks don’t reflect your domain. Set up an evaluation set with three types of cases: questions your system should answer correctly, out-of-scope questions it should reject, and trick questions (false premises) it should correct. Run that set every time you change temperature, prompt, model, or retrieval threshold.

Two minimum metrics: invention rate (responses with unsupported statements divided by responses given) and correct rejection rate (out-of-scope questions rejected divided by out-of-scope questions). If you optimize only the first, the system will end up rejecting everything; if you optimize only the second, it will answer by inventing. They move in tension and must be monitored together.

What this configuration doesn’t solve

Full transparency: even with everything above properly set up, residual hallucinations remain. The cases that are hardest to control are reasoning hallucinations in questions requiring several chained steps, those arising from ambiguous retrieved context (documents that contradict each other), and those that appear when the user pushes the model with persistent false premises. In regulated domains (health, legal, financial) no configuration replaces human review; the system can assist, but responsibility for the final statement cannot be delegated to the model.

The realistic expectation isn’t zero hallucinations, but hallucinations detected before causing harm, with a system that knows when to stay silent. That’s the configuration you can defend before a client or an auditor.