What a Hallucination Score Actually Measures

A vendor sends you a hallucination rate. It is a single number, it is low, and it sits in the assurance pack next to the accuracy figure and the latency figure as though it belongs there. The governance question that follows is almost always the wrong one, which is whether the number is low enough.

The right question is what the number is a measurement of. Because a hallucination score is not a property of a model. It is a property of a benchmark: one task, one definition, one dataset, scored by a detector that is itself a model with its own error rate. The leaderboards say this in their own documentation, usually a few paragraphs below the table.

This guide is about reading those numbers properly and about what to measure instead. It covers why the definition problem comes before the measurement problem, what the best known leaderboard actually tests, why published scores decay, the refusal confound that makes a cautious model look accurate, and the three tests that are yours rather than anyone else's. It sits under our guide to AI red-teaming and model testing, which owns testing method, and does not restate it.

The definition problem comes first

Before any measurement is meaningful, the thing being measured has to be defined, and this is where the field has not converged.

NIST avoids the popular word entirely. In the Generative AI Profile, which accompanies the AI Risk Management Framework, the risk is listed as "Confabulation: The production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived."

Read the fuller definition and the scope is wider than most measurement assumes. Confabulation "refers to a phenomenon in which GAI systems generate and confidently present erroneous or false content in response to prompts", and it "also include[s] generated outputs that diverge from the prompts or other input or that contradict previously generated statements in the same context".

That is three different failures under one heading.

• Factual error. The output states something untrue about the world.
• Divergence from input. The output asserts something the provided source does not support, whether or not it is true.
• Self contradiction. The output conflicts with what the same system said earlier in the same context.

No single score covers all three, because they need different ground truth. The first needs a reference about the world. The second needs only the source document. The third needs the conversation. Any number presented as "the hallucination rate" has silently picked one, and almost always the second, because it is the only one that is cheap to automate.

What the leaderboard actually measures

The clearest way to see the gap between a score and a specification is to read the method behind the most cited one. Vectara's hallucination leaderboard is well constructed, openly documented and honest about its scope, which is exactly why it makes the point.

The limitations are stated on the page, and they are the part to quote when someone presents the number as a specification. "We are not evaluating the quality of the summaries, only the factual consistency of them." And, plainly, "We do not claim to have solved the problem of hallucination detection."

The sharpest of them is this: an extractive summary that simply copies sentences from the source would score very well. Faithfulness to a source and usefulness to a reader are different properties, and a metric that rewards the first will rank a model that barely paraphrases above one that genuinely condenses. The page says the metric should be paired with other evaluations rather than used alone, which is a vendor telling you not to use their number the way most people use it.

And one scope note that matters for anyone using a model outside a retrieval pattern: hallucinations on open ended questions cannot be reliably checked, because most models' training data is unknown. If your use case is a model answering from its own knowledge rather than from a supplied document, the leaderboard is measuring a different thing from the thing you care about.

Why published scores decay

Benchmark numbers have a shelf life, and three mechanisms shorten it.

Definitions are not shared. The authors of the HalluLens benchmark put it directly: "A major challenge in benchmarking hallucinations is the lack of a unified framework due to inconsistent definitions and categorizations." Two benchmarks reporting a hallucination rate are frequently not reporting the same quantity, which makes cross-benchmark comparison meaningless and makes a vendor's choice of benchmark a selection decision.

Benchmarks saturate. The same paper analyses existing benchmarks "highlighting their limitations and saturation", and builds in dynamic test set generation specifically to mitigate data leakage. A public test set becomes training data eventually, and once it does the score measures familiarity rather than capability. This is why a benchmark that discriminated well last year can show every frontier model clustered at the top this year, which has happened to the leaderboards and prompted new versions.

The detector is itself a model. Automated scoring at this scale requires a judge model, which has its own failure modes and its own drift. A score produced by model A evaluating model B inherits uncertainty from both, and when the detector is updated the historical numbers are not strictly comparable to the new ones.

The refusal confound

This is the single most important thing to understand about these numbers and the easiest to miss, because it is usually in a separate column.

A model that declines to answer cannot produce an unsupported claim. So abstention improves a hallucination rate, and a model tuned to refuse more will look more reliable on the metric while being less useful. The leaderboard reports Answer Rate separately precisely so that this is visible, and filters refusals and very short answers out of the scoring.

Read in isolation, the hallucination rate rewards caution. Read alongside Answer Rate, it tells you something real: whether a model achieved its score by being accurate or by being quiet. Any assurance pack quoting the first without the second has removed the information that makes the first interpretable.

This also sets up the governance judgement that actually matters, which is not a threshold but a trade. For a clinical summarisation tool, abstention is usually the preferred failure and a high refusal rate is a feature. For a customer-facing assistant, a model that refuses a fifth of requests has moved the failure from an incorrect answer to an unresolved query, which is cheaper to govern and still a cost. Which failure you prefer is a decision about consequence, and our guide to risk tiering covers how that decision should follow from the tier.

Three tests that are yours to run

Published scores are useful for shortlisting a model and almost useless for assuring a system, because a system is a model plus a prompt plus retrieval plus scaffolding plus a population of real requests. Three tests close most of that gap and none of them requires a research team.

1. A grounded task set from your own corpus
   Fifty to two hundred real inputs from your domain, with the answer a competent human would give, reviewed by someone who knows the subject. Score on whether the output is supported by the source you supplied, which is the one judgement you can make without a reference about the world. This is the single highest value artefact in generative assurance and it is mostly annotation work rather than engineering. Keep it private, version it, and reuse it as the probe set for post-market monitoring.
2. Abstention behaviour on questions you cannot answer
   Include cases where the correct response is to decline: the source does not contain the answer, the question is outside scope, the premise is false. Measure what the system does. Many systems that score well on supported questions answer unanswerable ones confidently, and that behaviour is invisible in any metric computed only over answered items.
3. Leakage of what the system was given
   Test whether the system reveals its instructions, its retrieved context from other users' sessions, or data it was handed for a different purpose. This is adjacent to security testing and belongs in a governance assessment too, because the question of what the system discloses is a data protection question before it is an attack surface.
4. And one measurement most teams already have
   Where a human reviews output, the correction rate on a sample of real reviews is a direct measurement of output quality in production, on the real distribution, with expert judgement built in. It is noisier than a benchmark and it measures your system rather than somebody's test set.

Toxicity is a different measurement problem

Toxicity and unsafe-content measurement are often grouped with hallucination under "output quality", and the measurement problem is structurally different in a way that matters for how much weight a score can carry.

Factual support, at least in the grounded case, has an objective referent: either the source says it or it does not. Whether content is harmful depends on who receives it, in what context, and against whose standard. The same sentence can be clinical in a medical setting, abusive in a workplace, and unremarkable in a transcript being analysed. A classifier score cannot carry that context, and a threshold set centrally will be wrong in both directions across a portfolio of use cases.

The practical consequence is that toxicity thresholds should be set per application rather than per organisation, and that the artefact worth keeping is the list of categories that matter for this use case and the examples that define the boundary, rather than a single number. Where the content concerns protected characteristics, this overlaps with fairness measurement, and our guide to bias auditing and fairness testing covers that discipline.

What goes in the assessment record instead

If a single score is not the answer, the assessment still has to conclude something. Six things make a generative assurance record defensible, and five of them are not numbers.

• Which failure you are measuring. Factual error, divergence from supplied source, or self contradiction. Name it, because the test only covers one.
• The task set, versioned. What was tested, from where, reviewed by whom. Private and reusable.
• Results with the answer rate beside them. Never the quality figure alone.
• Abstention behaviour. What the system does when it should decline, measured separately.
• The accepted failure mode. A written statement of which failure this application prefers, and why, tied to the tier.
• Any published score, with its provenance. Which benchmark, which version, which detector, what date. Useful context, not a finding.

Our guide to audit ready evidence covers where records like this should live so that a result can be reproduced against the model and policy versions in force when it was produced.

Frequently asked questions

Where this leaves you

The reason a hallucination score reads like a specification is that it is formatted like one. A percentage, to one decimal place, in a table next to other percentages. Nothing about the presentation signals that it measures one task under one of three possible definitions, scored by a model, on a corpus you cannot inspect, with no stated validity period.

None of that makes the number dishonest. The leaderboards document their limits carefully and say outright that detection is not solved and that their metric should not be used alone. The failure is downstream, in governance programmes that accept the figure as a property of the model and file it as assurance. The work that produces real assurance is unglamorous and mostly annotation: a few dozen real cases from your own domain, with expert answers, run against the system you actually deployed, with the refusals counted separately.