Beyond Single Ground Truth: Reference Monism as Epistemic Injustice in ASR Evaluation

Authors: Anna Seo Gyeong Choi, Maria Teleki, James Caverlee et al.

arXiv · PDF

Summary

The authors argue that automatic speech recognition (ASR) systems commit "epistemic injustice" when they enforce a single transcription standard as ground truth. Different transcription conventions (verbatim vs. cleaned-up) produce different "correct" transcripts for the same speech, and speakers with aphasia—whose disfluencies carry clinical meaning—are systematically penalized when evaluated against "clean" references that treat those features as errors. The paper proposes WER-Range: reporting performance across multiple legitimate transcription conventions instead of assuming one right answer.

Main takeaways:

• Word Error Rate (WER) varies depending on which transcription convention you pick as ground truth—not because the system changed, but because "ground truth" itself is a choice.
• Speakers with aphasia are disadvantaged when their meaningful disfluencies are treated as errors to be removed from the reference transcript.
• The authors introduce Epistemic Injustice Distance (EID) to quantify the harm of enforcing a single standard.
• WER-Range proposes reporting ASR performance across multiple legitimate conventions rather than picking one.
• The problem isn't just differential performance—it's that the evaluation infrastructure lacks the conceptual tools to recognize some speech as legitimate in the first place.

Relevance

Not directly related to my persona or midtraining work—this is about speech recognition and social justice in evaluation. Potentially relevant if I ever think about how different "ground truth" conventions for assistant behavior (helpful vs. verbose vs. concise) might systematically advantage or disadvantage certain installation methods.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.

Abstract

arXiv:2605.07084v1 Announce Type: new Abstract: Automatic speech recognition (ASR) evaluation compares system output to ground truth transcripts, with Word Error Rate (WER) quantifying the distance between them. But ground truth transcripts are not discovered - they are produced by human annotators following conventions that encode normative assumptions about which speech features matter. Different conventions (verbatim, non-verbatim, legal) produce different transcripts of identical speech and judge the same ASR output differently. This paper argues that reference monism - enforcing a single transcription convention as ground truth - commits epistemic injustice. Speakers with aphasia, whose speech includes clinically meaningful disfluencies, are systematically disadvantaged when evaluated against "clean" references that treat those disfluencies as errors. The harm is not merely differential performance, but that evaluative infrastructure lacks interpretive resources to recognize their contributions as legitimate. We develop a philosophical framework introducing the hermeneutical gap, formalize Epistemic Injustice Distance (EID) to measure reference monism's cost, and demonstrate empirically using AphasiaBank that WER varies depending on which convention defines ground truth. We propose WER-Range: reporting performance across legitimate conventions rather than assuming a single correct answer.