AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment

Authors: Robin Linzmayer (Department of Computer Science, Columbia University, Department of Biomedical Informatics et al.

arXiv · PDF

Summary

The authors introduce AcuityBench, a benchmark that tests whether language models correctly judge how urgently a patient needs care (home monitoring, scheduled visit, urgent care, or ER) from medical descriptions. It pools 914 cases from five datasets (conversations, forum posts, clinical vignettes, patient messages), including 217 ambiguous cases where physicians themselves disagreed. They test 12 frontier models in both QA format (pick one of four urgency levels) and free-form conversational format, finding substantial variation in accuracy and a systematic tradeoff: conversational responses reduce over-triage but increase under-triage, especially for high-acuity cases. No model closely matches the distribution of physician uncertainty on ambiguous cases.

Main takeaways:

• AcuityBench unifies five public medical datasets under a shared four-level urgency framework (home monitoring → scheduled → urgent → ER) with 697 clear-consensus cases and 217 physician-confirmed ambiguous cases.
• Frontier models vary widely in accuracy and error direction; some over-triage, others under-triage.
• Conversational responses systematically reduce over-triage but increase under-triage compared to QA format, especially for high-acuity cases.
• No model's uncertainty distribution on ambiguous cases matches physician judgment distributions — models are more confident and concentrated.
• Acuity identification is a distinct safety-critical capability separate from general medical QA.

Relevance

Not related to my persona installation or conditional behavior research — this is a domain-specific medical safety benchmark. Included because it's a concrete example of task-format effects (QA vs conversational) shifting behavior systematically, which parallels my interest in installation-path equivalence.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.

Abstract

arXiv:2605.11398v1 Announce Type: new Abstract: We introduce AcuityBench, a benchmark for evaluating whether language models identify the appropriate urgency of care from user medical presentations. Existing health benchmarks emphasize medical question answering, broad health interactions, or narrow workflow-specific triage tasks, but they do not offer a unified evaluation of acuity identification across these settings. AcuityBench addresses this gap by harmonizing five public datasets spanning user conversations, online forum posts, clinical vignettes, and patient portal messages under a shared four-level acuity framework ranging from home monitoring to immediate emergency care. The benchmark contains 914 cases, including 697 consensus cases for standard accuracy evaluation and 217 physician-confirmed ambiguous cases for uncertainty-aware evaluation. It supports two complementary task formats: explicit four-way classification in a QA setting, and free-form conversational responses evaluated with a rubric-based judge anchored to the same framework. Across 12 frontier proprietary and open-weight models, we find substantial variation in clear-case acuity accuracy and error direction. Comparing task formats reveals a systematic tradeoff: conversational responses reduce over-triage but increase under-triage relative to QA, especially in higher-acuity cases. In ambiguous cases, no model closely matches the distribution of physician judgments, and model predictions are more concentrated than expert clinical uncertainty. We also compare expert and model adjudication on a subset of maximally ambiguous cases, using those cases to examine the role of clinical uncertainty in label disagreement. Together, these results position acuity identification as a distinct safety-critical capability and show that AcuityBench enables systematic comparison and stress-testing of how well models guide users to the right level of care in real-world health use.