DiffScore: Text Evaluation Beyond Autoregressive Likelihood

Authors: Wen Lai, Yingli Shen, Dingnan Jin et al.

arXiv · PDF

Summary

The authors argue that evaluating text with standard autoregressive language models introduces positional bias—early tokens are scored with less context than later ones—and propose DiffScore, which uses masked diffusion models to score every token with full bidirectional context. By measuring how well text can be reconstructed at different masking rates, DiffScore creates a quality hierarchy from local fluency to global coherence and provides diagnostic tools like multi-timestep profiles and bidirectional PMI decomposition that separate fluency from faithfulness.

Main takeaways:

• Autoregressive evaluation gives early tokens less context, conflating architectural asymmetry with actual text quality
• DiffScore uses masked reconstruction at varying masking rates, so every token is scored with full bidirectional context
• Multi-timestep profiles show how quality changes across masking rates, revealing whether problems are local (fluency) or global (coherence)
• Bidirectional PMI decomposition separates fluency (word choice) from faithfulness (semantic accuracy)
• Outperforms autoregressive baselines across ten benchmarks in both zero-shot and fine-tuned settings

Relevance

Not directly related to my persona or conditional behavior work—this is about text evaluation methodology. Could be useful if I need better tools to evaluate whether marker-implanted models produce qualitatively different outputs, but it's tangential to my core research on behavior installation mechanisms.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses bias, evaluation, benchmark.

Abstract

arXiv:2605.11601v1 Announce Type: new Abstract: Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only leftward context, conflating architectural asymmetry with true text quality. We propose masked reconstruction as an alternative paradigm, where every token is scored using full bidirectional context. We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models. By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias and naturally establishes an evaluation hierarchy from local fluency to global coherence. We further provide diagnostic tools unavailable to autoregressive frameworks: multi-timestep quality profiles that decompose scores across masking rates, and bidirectional PMI decomposition that disentangles fluency from faithfulness. Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings. The code is released at: https://github.com/wenlai-lavine/DiffScore.