Human-LLM Dialogue Improves Diagnostic Accuracy in Emergency Care
Authors: Burcu Sayin, Ngoc Vo Hong, Ipek Baris Schlicht et al.
Summary
The authors built MedSyn, a system where emergency physicians iteratively query an LLM that has the full clinical record while the physician initially sees only the chief complaint. Seven physicians (three seniors, four residents) completed baseline and AI-assisted sessions on 52 MIMIC-IV cases. Blinded evaluation showed residents' correctness on hard cases rose from 58.9% to 73.4%; standardized metrics confirmed medium effect size. Dialogue analysis revealed seniors asked targeted questions while residents asked broader queries, and cross-expertise agreement increased.
Main takeaways:
- Physicians query an LLM with full clinical info while they see only the chief complaint, then iteratively reveal details
- Residents' hard-case correctness improved from 58.9% to 73.4%; difficulty-standardized completely-correct rate showed medium effect (d=0.47)
- Automated metrics: standardized any-match accuracy +15.6 pp, residents' F1 +13.8 pp
- Seniors asked hypothesis-driven questions; residents used broader queries
- Cross-expertise diagnostic agreement increased by 14.5 pp
- Demonstrates measurable benefit in live physician workflows, not just benchmarks
Relevance
Not directly related to my persona or behavioral installation work. This is an applied study of LLMs as interactive decision-support tools in medicine, focused on workflow integration and diagnostic accuracy rather than how behaviors or personas are represented in the model.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.
Abstract
arXiv:2605.08533v1 Announce Type: new Abstract: Clinical decision-making in emergency medicine demands rapid, accurate diagnoses under uncertainty. Despite benchmark progress, evidence for LLMs as interactive aids in live physician workflows remains sparse. MedSyn lets physicians iteratively query an LLM provided with the full clinical record while initially viewing only the chief complaint. Seven physicians (three seniors, four residents) completed baseline and AI-assisted sessions across 52 MIMIC-IV cases stratified by difficulty. Blinded evaluation showed residents' Hard-case correctness rose from 0.589 to 0.734; difficulty-standardised completely-correct rates confirmed a medium effect ({\Delta} = 0.092; p = 0.071; d = 0.47). Automated metrics corroborated these gains: standardised any-match accuracy improved by 0.156 (p < 0.0001), and residents showed the largest F1 gain ({\Delta} = 0.138; p < 0.0001). Dialogue analysis revealed expertise-dependent strategies (seniors asked targeted, hypothesis-driven questions; residents relied on broader queries) and cross-expertise concordance increased ({\Delta} = 0.145; p < 0.0001). Interactive LLM support meaningfully enhances diagnostic reasoning.