ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV
Authors: Alex Stinard
Summary
ClinicalBench tests whether retrieval systems can handle the messy realities of clinical notes—negation ("no fever"), temporality (past vs present), and attribution (family history vs patient symptoms)—before feeding information to a reasoning model. The authors build a system (EpiKG) that tags every fact with assertion labels and routes retrieval by question intent, then benchmark it on 400 questions over real MIMIC-IV patient records. Their assertion-aware retrieval improves exact-match accuracy by 22 percentage points over a standard dense retrieval baseline, with physician adjudication confirming the gains.
Main takeaways:
- Clinical reasoning benchmarks usually test on clean inputs, but real EHR retrieval requires handling negation, time, and attribution
- Assertion-aware knowledge-graph retrieval (tracking "negated" vs "affirmed" and "past" vs "present") beats dense embedding retrieval by +22 percentage points
- The gain is larger (+39.5pp) on questions where keyword matching is deterministic, smaller on ambiguous cases
- Larger language models benefit less from better retrieval (beta=-1.12), possibly because they can compensate for retrieval errors
- Physician review found 56% of auto-generated reference answers were defective, highlighting evaluation challenges in clinical NLP
Relevance
Not directly related to my persona/midtraining work—included because it's a rigorous example of how context representation (here, assertion structure in clinical facts) shapes task performance, analogous to how persona/system-prompt structure might shape behavior installation.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.11143v1 Announce Type: new Abstract: Reasoning benchmarks measure clinical performance on clean inputs. We evaluate the step before reasoning: retrieval over real EHR notes, where negation, temporality, and family-versus-patient attribution can flip a correct answer to a wrong one. EpiKG carries an assertion label and a temporality tag with every fact in a patient knowledge graph, then routes retrieval by question intent. ClinicalBench is a 400-question test over 43 MIMIC-IV patients across 9 assertion-sensitive categories. A 7-condition ablation tests each piece of EpiKG across six LLMs (Claude Opus 4.6, GPT-OSS 20B, MedGemma 27B, Gemma 4 31B, MedGemma 1.5 4B, Qwen 3.5 35B). Three physicians blindly adjudicated 100 paired items. The author-blind primary endpoint, leave-author-out paired exact McNemar on 50 unanimous-strict items rated by two external physicians, yields +22.0 percentage points (95 percent Newcombe CI [+5.1, +31.5], p=0.0192). The architectural novelty, intent-aware KG-RAG over a Contriever dense-RAG baseline (C2b to C4g_kw on the change-excluded n=362 endpoint), is +8.84 percentage points (paired McNemar p=1.79e-3); +12.43 percentage points under oracle intent. Sensitivities agree directionally: three-rater physician majority +24.0 percentage points (subject to single-author circularity); deterministic keyword reproducibility proxy +39.5 percentage points. Across the six models, the gain shrinks as the LLM-alone baseline rises (beta=-1.123, r=-0.921, p=0.009). With n=6 this looks more like regression to the mean than encoding substituting for model size. Physician adjudication identified 56 percent of auto-generated reference answers as defective, a methodological finding indicating that NLP-pipeline clinical-QA benchmarks require physician adjudication to be usable. ClinicalBench, the frozen evaluator, three-rater adjudication data, and the EpiKG output stack are publicly released.