Hallucination Detection in Production: RAG Grounding Checks, Self-Consistency, and Guardrails

Why you cannot eliminate hallucinations (only reduce them)

An LLM generates text by sampling from a probability distribution over next tokens. Sometimes the highest-probability continuation is factually wrong but stylistically fluent — the model has no internal sense of "I don't know". Even models trained with RLHF and factuality tuning hallucinate on roughly 3-8% of factual queries in open-domain benchmarks, and higher on specialized or time-sensitive topics. No prompt engineering or model selection gets this to zero.

The goal of hallucination detection is therefore reduction to an acceptable rate, not elimination. For casual chat, 5% fabrication might be tolerable; for medical or legal advice, even 1% is dangerous. The layers below let you dial the detection sensitivity — and the cost — to match your domain's tolerance.

Layer 1: RAG grounding checks

The most common production hallucination in RAG systems is the model ignoring the retrieved context and answering from its parametric memory — which may be outdated or wrong. A grounding check verifies that every factual claim in the answer is supported by the retrieved source chunks. The standard implementation uses LLM-as-a-Judge: a second LLM call takes the answer plus the source chunks and labels each claim as supported, contradicted, or not_mentioned.

# RAG grounding check (LLM-as-judge)
JUDGE_PROMPT = """Given the SOURCE DOCUMENTS and the ANSWER,
label each factual claim in the ANSWER as:
- supported: directly stated in the sources
- contradicted: contradicted by the sources
- not_mentioned: the sources do not address this claim

Return JSON: {"claims": [{"claim": "...", "label": "..."}]}

SOURCES:
{retrieved_chunks}

ANSWER:
{generated_answer}"""

# If any claim is "contradicted" → reject and regenerate
# If claims are "not_mentioned" → flag for human review

Grounding checks catch the "model invented a fact not in the documents" failure mode, which is the dominant hallucination pattern in RAG. They add one LLM call per answer (typically to a cheaper model like GPT-4o-mini, since judging is easier than generating), adding 200-800ms and a few cents per thousand queries. For any RAG system serving factual content, this layer is non-optional.

Layer 2: self-consistency sampling

Self-consistency exploits a simple signal: if an LLM is confident and correct, sampling the same prompt multiple times at temperature > 0 usually yields the same answer. If it is hallucinating, the samples diverge. The method samples N responses (typically 3-5) and takes a majority vote or measures agreement. High agreement = high confidence; low agreement = the answer is uncertain and should be flagged or escalated.

The cost is N× the LLM calls, which makes self-consistency impractical for every request. The right pattern is to apply it selectively: to high-stakes factual queries (medical, legal, financial) or to queries where the grounding check failed. For a workload where 10% of queries are high-stakes, the blended cost overhead is roughly 0.1 × (5-1) = 0.4, or 40% — significant but contained.

Layer 3: LLM-as-a-judge factuality

Beyond grounding (which checks against retrieved sources), an LLM judge can check an answer against an external knowledge base, a set of reference facts, or common-sense reasoning. This is broader than grounding — it catches hallucinations even when no retrieval is involved. The judge prompt asks "is this answer factually accurate? List any errors." and the output gates whether the answer reaches the user.

The trade-off is cost and latency: the judge is another LLM call. For most workloads, you do not run a judge on every query. Instead, you run it asynchronously on a sample (1-5% of traffic) to measure the hallucination rate over time, and you run it synchronously only on high-stakes or low-confidence queries. This is the LLM-as-a-Judge methodology applied specifically to factuality — see that guide for the judge-prompt patterns and calibration.

Layer 4: guardrails

Guardrails are the structural last line of defense — deterministic or model-based filters that catch specific hallucination patterns before the output reaches the user. Two open-source frameworks dominate.

# Guardrails AI — factuality validator on structured output
from guardrails import Guard
from guardrails.hub import FactCheckValidator

guard = Guard().use(
    FactCheckValidator, on="fail", action="reask"
)

# The guard validates the LLM output against a factuality check
# and re-asks the model if the check fails
response = guard(
    llm_api=litellm.completion,
    model="gpt-4o",
    prompt="What is the capital of France?",
    output_schema={"capital": "string"}
)
# If the model says "Paris" → passes. If it says "London" → reask.

Guardrails are best for structured outputs (JSON with expected fields) and for enforcing topic boundaries. They are less effective for free-form chat, where the hallucination patterns are too varied for deterministic rules. Pair guardrails with the softer layers (grounding, judging) rather than relying on them alone.

The layered defense: which layers, when

No single layer catches all hallucinations. The practical production setup combines them based on query stakes and workload type.

Monitoring hallucination rate over time

Detection layers deployed in production generate the data to track hallucination rate as a metric. The dashboard you want is: percentage of queries flagged by each layer (grounding-rejected, self-consistency-low, judge-failed) over time. If the grounding-rejection rate spikes from 3% to 8% after a model upgrade, the new model is hallucinating more — roll back or adjust the prompt. If it trends down after a prompt-engineering change, the change worked.

Connect these metrics to your agent traces so you can slice hallucination rate by feature, model, and query type. The combination of trace-level observability and hallucination-specific metrics is what turns "we think the model is pretty good" into "our factual hallucination rate is 2.3% on RAG queries, down from 4.1% last quarter."

FAQ

Can you fully eliminate LLM hallucinations?

No — LLMs are probabilistic and will occasionally fabricate. The goal is reduction to an acceptable rate (under 1-5% depending on domain). Combine grounding, verification, and guardrails to reach the rate your quality bar requires. Higher-stakes domains justify more (and more expensive) detection layers.

What is a RAG grounding check?

A verification that every factual claim in the LLM's answer is supported by the retrieved documents. Typically implemented as an LLM-as-judge that labels each claim as supported, contradicted, or not-mentioned. Answers with unsupported claims are rejected or regenerated. This catches the dominant RAG failure: the model ignoring retrieved context and answering from parametric memory.

Does self-consistency reduce hallucinations?

Yes, for factual and reasoning tasks, by sampling N responses and taking a majority vote. High agreement signals confidence; low agreement signals likely hallucination. The cost is N× LLM calls, so it is reserved for high-stakes queries. Note: it detects uncertainty, not falsity — a confidently-wrong model hallucinates consistently, so pair it with grounding.

NeMo Guardrails vs Guardrails AI — which to use?

NeMo Guardrails (NVIDIA) is conversation-flow-centric — best for constraining dialog and blocking off-topic inputs. Guardrails AI is validation-centric — best for structurally validating outputs (schema, regex, factuality). For pure hallucination detection on factual outputs, Guardrails AI or a custom LLM-as-judge is usually simpler.

Related deep dives

• LLM-as-a-Judge — the methodology behind grounding checks and factuality judging
• Tracing LLM Agents — how to trace which step introduced a hallucination
• LangSmith vs Langfuse vs Phoenix — platforms that store hallucination-detection signals alongside traces
• Cost Monitoring Dashboards — the cost that detection layers add, tracked alongside spend