DARE: Diffusion Language Model Activation Reuse for Efficient Inference

Authors: Natalia Frumkin, Bokun Wang, Hung-Yueh Chiang et al.

arXiv · PDF

Summary

The authors identify that diffusion language models (an alternative to autoregressive LMs) have redundant attention activations across tokens during inference. They built DARE, which reuses cached key-value pairs (DARE-KV) and output activations (DARE-O) when the model's internal state predicts they won't change much. This achieves up to 1.20x per-layer speedup and reuses up to 87% of attention activations with minimal quality loss (around 2% average performance drop). The technique is compatible with other speedup methods like prefix caching.

Main takeaways:

• Diffusion LLMs show high token-wise redundancy in bi-directional self-attention activations
• DARE reuses cached key-value and output activations when temporal changes in queries predict redundancy
• Achieves up to 1.20x per-layer latency reduction, reusing up to 87% of attention activations
• Average performance drops are only 2.0% (DARE-KV) and 1.2% (DARE-O) on reasoning and code benchmarks
• Compatible with other inference optimizations like prefix caching and Fast-dLLM

Relevance

Not directly related to my persona or midtraining work—this is an inference optimization for diffusion language models. Could be tangentially relevant if I ever investigate how persona representations propagate through attention layers during generation.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.

Abstract

arXiv:2605.08134v1 Announce Type: new Abstract: Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to auto-regressive (AR) models, offering greater expressive capacity and potential for parallel generation and faster inference. However, open-source dLLMs remain immature, lagging behind AR models in both efficiency and quality. We identify an underexplored property of dLLMs: token-wise redundancy in bi-directional self-attention. Self-attention activations are highly correlated across tokens, and temporal changes in query representations can predict redundancy in corresponding key, value, and output activations. We introduce DARE, with two complementary mechanisms: DARE-KV, which reuses cached key-value (KV) activations, and DARE-O, which reuses output activations to reduce redundant computation while preserving quality. DARE achieves up to 1.20x per-layer latency reduction and reuses up to 87% of attention activations, with negligible degradation on reasoning and code-generation benchmarks. DARE-KV and DARE-O incur average performance drops of only 2.0% and 1.2%, respectively. Combined with techniques such as prefix caching and Fast-dLLM, DARE provides additive gains without retraining. These results establish token-wise reuse as an effective strategy for improving the efficiency of diffusion-based LLMs while preserving generation fidelity. Code: https://github.com/enyac-group/DARE