Detecting natural hallucinations in local RAG

Span detectors, LLM judges, and the limits of ensembling

Research

MSc AI dissertation

Institution

Queen Mary University of London

Supervisor & year

Syed Rafee ·

2026

Read the paper (PDF, opens in a new tab)

The evaluation set

345 claims, every one checked by hand

Supported claims303
Hallucinated42

Every claim the system made, coloured by its hand-checked label.

27 partly supported · 15 unsupported

The study

Grounding answers in documents does not stop a language model from inventing things, so I tested whether today’s hallucination detectors catch the mistakes a careful, fully local RAG system makes on its own.

I built a retrieval-augmented question-answering system that runs entirely on one laptop, over six UK government guidance documents, the “coloured books” such as HM Treasury’s Green Book. Its answers to 100 test questions broke down into 346 short claims, and I labelled every one against the passages it was based on: green when they support it, yellow when they only partly do, red when they do not.

Five ways of checking those claims were then compared on exactly the same answers: a span detector trained on the RAGTruth benchmark, an LLM judge from three model families, and three ways of combining the two. Everything ran on 7 to 9 billion parameter models, the setting open to a government team that cannot send its documents to an external API.

Contributions

  • A hand-labelled set of 345 claims from 100 questions
  • Five verification systems compared on frozen answers
  • Three judge families, the third pre-registered
  • Why a benchmark-trained span detector misses natural errors
  • Judge families catch different errors: union beats agreement
  • The cost and the blind spot of a screening cascade

In numbers

12.2%

of the claims were hallucinations, 42 of 345, mostly subtle and mostly in ordinary questions

40/42

hallucinated claims got no span at all from the benchmark-trained detector

38.1%

F1 when two judge families and the span detector flag together, against 26.1% for the best single judge

0/10

real errors among the claims all three judges flagged together

What I asked, and what I found

Four research questions, tested on answers that were generated once and then frozen, so every detector judged exactly the same claims.

  1. 01

    How often, and where, does a well-built local RAG system hallucinate?

    In 12.2% of its claims, and 33 of those 42 come from ordinary answerable questions: correct facts joined the wrong way, “should” turned into “must”, one document’s words given to another.

  2. 02

    How well do a span detector and an LLM judge catch these natural errors?

    Barely. The span detector flags 2 of 42, near chance. Each 8B judge catches 6 to 9, with at best one true alarm in three.

  3. 03

    Do combinations of the two do better than either alone?

    Not the ones designed: a silent member pulls every rule down to the judge alone. What helps is a union of judges from different families.

  4. 04

    What do the trade-offs between cost and accuracy look like?

    A screening cascade cuts LLM calls by 99%, from 345 to 2, and catches almost nothing, because a cascade can never see more than its screener.

How it works

A fully local pipeline: the documents are indexed once, every question is answered once and frozen, and then each verifier judges exactly the same claims against the same passages.

Ingestion

offline, once

  1. Corpus

    6 documents · 721 pages

  2. Cleaning

    pdfplumber · audited page by page

  3. Chunking

    heading-aware · 1,000 / 150

  4. Index

    2,473 chunks · bge-large-en-v1.5 in Chroma, and BM25

Answering

one pass per question, then frozen

  1. Questions

    100 · five types

  2. Hybrid retriever

    dense 20 ∪ BM25 20

  3. Reranker

    bge-reranker-v2-m3 · top 5

  4. Generator

    Llama-3.1-8B · temperature 0

    LLM

  5. Claim extractor

    Qwen2.5-14B · two guards

    LLM

  6. 346 claims

    with their top-5 passages

Verification

every claim, three judge runs

  1. Claim

    + top-5 passages + question

  2. S1 · S2

    span detector · LLM judge

    LLM

  3. S3 · S4 · S5

    AND · vote · cascade

  4. Verdict

    green · yellow · red

A traffic light for every claimJudged only against the five passages retrieved for its question, never against the web or what is true elsewhere.

Supported

303 claims

Stated in the passages, or follows directly from them.

Green · Pass

Partly supported

27 claims

A true core with an ungrounded addition, overstated certainty or the wrong source.

Yellow · Human review

Unsupported

15 claims

Absent from the passages, or contradicted by them.

Red · Block

Five ways to check a claimEach maps a claim and its passages to green, yellow or red. A claim counts as flagged when it comes back yellow or red.

S1

Span detector

LettuceDetect

A ModernBERT token classifier trained on RAGTruth. The share of the claim it marks as unsupported sets the label.

0 LLM calls
S2

LLM judge

Llama · Qwen · Gemma

Reasons first, then labels the claim against its passages, in JSON. One run per model family.

1 call a claim
S3

AND

S1 + S2

Both run, and the worse label wins.

1 call a claim
S4

Vote

S1 + S2

Agreement keeps the label; any disagreement goes to human review.

1 call a claim
S5

Cascade

S1, then S2

The span detector screens every claim; only what it flags reaches the judge.

0.006 calls a claim

What I found

Seven results, from what the errors look like to what it costs to catch them. Every chart is drawn from the dissertation’s own tables and figures.

01

The errors are correct facts, joined the wrong way

I coded all 42 errors by hand. The most common is wrong attachment: every fact is in the passages, but they are put together wrongly. One answer attached the Teal Book’s risk-response definitions to the wrong categories, five red claims in which every word is supported.

Next come overstatements, a “should” turned into a “must”, and one document’s wording given to another. 24 of the 42 were missed by every system under every judge.

Wrong attachment

Every fact is in the passages, joined the wrong way.

11 unsupported and 5 partly supported, 7 of 16 caught

Overstated certainty or scope

“Should” becomes “must”, or the claim reaches further than the source.

0 unsupported and 11 partly supported, 5 of 11 caught

Cross-document blending

One document’s wording given to another.

1 unsupported and 5 partly supported, 2 of 6 caught

Premise echoes

A trap question’s false premise, repeated as fact.

1 unsupported and 4 partly supported, 2 of 5 caught

Numbers and lists

A formula, figure or enumeration corrupted.

2 unsupported and 1 partly supported, 1 of 3 caught

Meta-claims

A claim about what the sources say.

0 unsupported and 1 partly supported, 1 of 1 caught
The 42 claims labelled yellow or red, coded by hand into six kinds. Caught: flagged by at least one of the five systems under any of the three judges.

02

The span detector is almost blind

LettuceDetect marked a span in only 2 of the 345 claims, both real errors from a single question. The other 40 errors got no span at all, so no threshold can rescue it: its AUROC is 0.524, close to a coin toss.

It was trained on the more frequent mistakes of earlier, less reliable generators. When correct facts are joined the wrong way, no single word is wrong for it to mark.

Claims the span detector flags at each threshold, of 42 hallucinated claims
ThresholdClaims flagged
0.0012
0.052
0.12
0.22
0.3 (used)2
0.52
0.71
0.90
Claims the span detector flags as its confidence threshold rises, against the 42 it should find. The systems use 0.3. From the sweep in the dissertation’s appendix B.

AUROC

0.524

Near chance: a coin toss scores 0.5

Precision

100%

Both of its flags are real errors

Recall

4.8%

2 of the 42 errors, from one question

03

Every judge is weak, and weak in its own way

Each 8B judge flags 25 to 36 claims and is right one time in three at best. Qwen does best, catching 9 of the 42; Gemma, the pre-registered third family, flags the most and is right least often.

Pairing a judge with the span detector changes almost nothing. With only two flags of its own, AND and vote collapse into the judge alone, and the cascade inherits the detector’s blindness.

Flag-task F1 of each system under each judge
SystemLlama-3.1-8BQwen2.5-7BGemma-2-9B
S1 Span detector9.1%9.1%9.1%
S2 LLM judge17.9%26.1%17.9%
S3 AND20.6%31.0%20.3%
S4 Vote20.6%31.0%20.3%
S5 Cascade4.7%0%4.7%
F1 on the flag task: a claim counts as flagged when it comes back yellow or red, and the 42 claims labelled yellow or red are the ones to find. The span detector works without a judge.

04

Different families catch different errors

Llama and Qwen never catch the same error: 6 and 9 catches, none shared. Yet 11 of their false alarms are the same claims. Gemma, run last, mostly catches what Llama already caught, 4 of its 7.

No error was caught by all three judges, and all 10 claims the three flag together are false alarms.

True alarms

Llama · 6Qwen · 9Gemma · 72820410

25 of the 42 errors flagged by no judge

False alarms

Llama · 19Qwen · 18Gemma · 29541313310

264 of the 303 supported claims flagged by no judge

Each circle is one judge family and each number counts the claims in its region; a judge’s total is beside its name. From the dissertation’s figure 2.

05

So flag what any judge flags

Because their catches differ and their false alarms repeat, the union of Llama and Qwen reaches 36.1% F1, 1.4 times the better judge alone, and the span detector’s extra catch lifts it to 38.1%. Requiring agreement does the opposite: it keeps exactly the shared mistakes.

This was found after seeing the results, so it needs confirming on new data.

F1 of each way of combining the judges
CombinationErrors caughtFalse alarmsF1
Llama or Qwen or span detector162638.1%
Llama or Qwen152636.1%
Any of the three judges173934.7%
Qwen alone91826.1%
Llama alone61917.9%
Gemma alone72917.9%
Two of the three agree51715.6%
Llama and Qwen agree0110%
All three agree0100%
F1 of each way of combining the judges, best first. Exploratory: the combinations were analysed after the results were in.

06

Most false alarms are claims about the sources

23 claims say what the passages do or do not mention, such as “the passages do not mention X”. A judge looks for a passage that supports the claim, and an absence has none, so it flags them even when they are right.

Set them aside and Llama’s and Qwen’s precision nearly doubles. Their wording gives them away, so they can go to a check of their own.

Precision with and without the meta-claims
JudgeEvery claimWithout meta-claims
Llama24.0%41.7%
Qwen33.3%69.2%
Gemma19.4%27.3%
Llama or Qwen36.6%58.3%
Precision with every claim, and with the 23 meta-claims set aside. A post-hoc analysis.

Llama’s false alarms

12/19

were meta-claims

Qwen’s false alarms

14/18

were meta-claims

Shared false alarms

10/11

were meta-claims

07

A cheap screen sees almost nothing

Letting the span detector screen every claim, and sending only what it flags to a judge, cuts LLM calls from 345 to 2 and the time per claim from 5–7 seconds to under 2.

But a cascade never sees more than its screener. It catches 1 error under Llama and Gemma and none under Qwen, whose judge overturned both of the screener’s correct flags.

Judge on every claimCascade
LLM calls99% fewer3452
Time per claimdepending on the judge5–7 s0.7–1.9 s
Errors caughtof the 42up to 110–1
Errors caught are out of 42: up to 11 for AND and vote, which run the judge on every claim, and 1 for the cascade under Llama or Gemma, 0 under Qwen.

What it means

For a team putting a local RAG system in front of people, four things to take from it.

01

Validate on your own traffic

Do not deploy a detector on its benchmark numbers. This one went from benchmark-competitive to near chance on this corpus.

02

Combine judges by union, not agreement

Use judges from different families and flag what any of them flags, or send disagreements to review. Requiring agreement caught nothing.

03

A cascade is only as good as its screener

Its recall is capped by the screener, and its referee can still overturn the screener’s correct catches.

04

Give meta-claims their own check

Claims about what the sources do not say are easy to spot by their wording, and they caused most of the judges’ false alarms.

Limitations

  • Only 42 real errors, so some differences cannot be resolved statistically.
  • 12.2% is the rate in a deliberately varied test set, not in everyday use.
  • The union and the meta-claim analysis are post hoc and need confirming on new data.
  • One annotator decided every label, with hard cases discussed with two reviewers.
  • One corpus, one generator and one claim extractor.

What comes next

  • Larger local judges, 27B to 32B, to see whether the weakness is the 8B scale.
  • A judge prompt that knows how to verify a claim of absence, tested under pre-registration.
  • Induced hallucinations, closer to what the span detector was trained on.
  • A cheap way to estimate in advance how much two judges overlap.
  • Answer-level correctness, and when the system should refuse to answer.

Read the full paper

The whole dissertation in 11 pages: the method in full, every table and figure, the statistical tests, the predictions made before the third judge was run, and the appendices.

MSc AI dissertation · Queen Mary University of London · 2026