The Translation Tax Is Not a Scalar: A Counterfactual Audit of English-Source Cue Inheritance in Chinese Multilingual Benchmarks
Authors: Zezheng Lin, Fengming Liu, Handi Li
Summary
Translated benchmarks are assumed to inflate scores by preserving English-source cues (the "Translation Tax"), but the authors show this isn't a simple scalar effect when auditing English-to-Chinese benchmarks. Three different measurement approaches disagree: back-translation gaps are small, cue-score calibration doesn't predict item-level gains, and native-control comparisons show model-family effects rather than uniform inflation. An LLM-naturalization stress test (rewriting Chinese surface form while keeping content fixed) reveals item-dependent validity risks rather than a single translation tax—high-residue items benefit from naturalization, low-residue items don't.
Main takeaways:
- The "Translation Tax" (score inflation from English cues in translated benchmarks) isn't a uniform scalar effect
- Three proxy estimators give inconsistent results: back-translation, cue-score calibration, and native-control comparisons
- LLM naturalization (rewriting surface form while preserving content) shows item-dependent effects: some items benefit, others don't
- Translation validity risk depends on estimator choice, item properties, and model family, not a single benchmark-wide factor
Relevance
Not directly related to my persona/midtraining work—this is about multilingual benchmark validity. Only tangentially relevant if I ever need to think about how surface-form variations (like translation artifacts) interact with installed behaviors or if prompt reformulation affects marker implantation.
Threat model
Potential threat/caveat for clean result "Stretching turn count, completion length, or system-prompt length at train time fails to amplify marker uptake; the longest system prompt instead leaks across bystander personas (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.07093v1 Announce Type: new Abstract: The Translation Tax is often treated as a scalar: translated benchmarks are assumed to inflate scores by preserving English-source cues. We audit this claim in an English-to-Chinese setting. Three proxy estimators disagree: back-translation gaps are small and parser-fragile; cue-score calibration does not predict item-level gains; and a six-model native-control comparison shows model-family rather than uniform benchmark effects. We add a same-item LLM-naturalization stress test that holds answer, options, and content fixed while rewriting Chinese surface form. After correcting a prompt-construction bug, this contrast no longer supports a model-family interaction, but it preserves a residue dose-response: high-residue items benefit while low-residue items do not. The result is not a single Translation Tax, but a set of estimator- and item-dependent validity risks. We release per-cell evidence, the naturalization protocol, human QC, and a reporting checklist for translated multilingual benchmark papers.