EPS
← All batches·2605.11330

Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights

topic: general_safetytop score: 100released: 2026-05-13first surfaced: 2026-05-13arXivPDFthreats2026-05-13

Authors: Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi et al.

arXiv · PDF

Summary

The authors argue that existing benchmarks for testing hallucination detectors fall short—they lack long-context RAG examples and don't simulate realistic label noise. They build TRIVIA+, a new benchmark with the longest context in the literature, human-annotated answers, and four different flavors of noisy labels (both sample-dependent and sample-independent). Testing popular hallucination detectors on TRIVIA+ reveals that current methods have a long way to go, basic LLM-as-a-judge baselines are surprisingly competitive, and label noise hurts performance.

Main takeaways:

  • Existing hallucination benchmarks don't test detectors on long RAG contexts or under realistic label noise, limiting their usefulness.
  • TRIVIA+ is a new RAG-based benchmark with the longest contexts available and four curated noise schemes for stress-testing detectors.
  • Current state-of-the-art detectors leave significant headroom for improvement on RAG tasks.
  • Simple LLM-as-a-Judge baselines perform competitively with more complex methods.
  • Label noise—whether from human annotators or automated labeling—degrades detector performance, highlighting robustness gaps.

Relevance

Not directly related to my persona/midtraining work—included because it's a useful benchmark if I ever need to evaluate whether conditional behaviors (e.g., personas) introduce hallucination or unfaithful outputs in RAG settings.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.

Abstract

arXiv:2605.11330v1 Announce Type: new Abstract: Hallucination, broadly referring to unfaithful, fabricated, or inconsistent content generated by LLMs, has wide-ranging implications. Therefore, a large body of effort has been devoted to detecting LLM hallucinations, as well as designing benchmark datasets for evaluating these detectors. In this work, we first establish a desiderata of properties for hallucination detection benchmarks (HDBs) to exhibit for effective evaluation. A critical look at existing HDBs through the lens of our desiderata reveals that none of them exhibits all the properties. We identify two largest gaps: (1) RAG-based grounded benchmarks with long context are severely lacking (partly because length impedes human annotation); and (2) Existing benchmarks do not make available realistic label noise for stress-testing detectors although real-world use-cases often grapple with label noise due to human or automated/weak annotation. To close these gaps, we build and open-source a new RAG-based HDB called T RIVIA+ that underwent a rigorous human annotation process. Notably, our benchmark exhibits all desirable properties including (1) T RIVIA+ contains samples with the longest context in the literature; and (2) we design and share four sets of noisy labels with different, both sample-dependent and sampleindependent, noise schemes. Finally, we perform experiments on RAG-based HDBs, including our T RIVIA+, using popular SOTA detectors that reveal new insights: (i) ample room remains for current detectors to reach the performance ceiling on RAG-based HDBs, (ii) the basic LLM-as-a-Judge baseline performs competitively, and (iii) label noise hinders detection performance. We expect that our findings, along with our proposed benchmark 1 , will motivate and foster needed research on hallucination detection for RAG-based tasks.