Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models
Authors: Hanhan Zhou, Shamik Roy, Rashmi Gangadharaiah
Summary
The authors study controlled text generation in discrete diffusion language models (DLMs), which generate text by denoising all positions in parallel rather than left-to-right. They find that applying steering interventions uniformly across all denoising steps degrades quality, especially when steering multiple attributes at once. Using sparse autoencoders trained on four DLMs, they discover that different attributes (topic, sentiment, etc.) "commit" at different points in the denoising schedule — topic solidifies in the first 2% of steps, while sentiment emerges gradually over 20%. They propose an adaptive scheduler that concentrates interventions only when each attribute is actively forming.
Main takeaways:
- Uniform steering at every denoising step in DLMs wastes effort on timesteps where the target attribute has already solidified or hasn't emerged yet, degrading generation quality.
- Different attributes commit on distinct schedules: topic commits early (first 2% of denoising), sentiment commits gradually (over 20%).
- An adaptive scheduler that intervenes only during attribute formation achieves up to 93% steering strength on three-attribute control, beating the best baseline by 15 points while preserving quality.
- Sparse autoencoders trained on DLMs reveal when and how strongly different attributes emerge during generation.
- The advantage of adaptive over uniform scheduling is governed by a single "dispersion statistic" of the commitment distribution, giving a closed-form cost-control trade-off.
Relevance
Related to my installation-path equivalence work: this shows that steering interventions have timing-dependent effects in diffusion models, analogous to how I might find that activation steering or fine-tuning operates at different layers or stages in autoregressive models; the sparse-autoencoder diagnostics for attribute commitment could inspire similar diagnostics for when persona behaviors "commit" during generation.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses failure.
Abstract
arXiv:2605.10971v1 Announce Type: new Abstract: Discrete diffusion language models (DLMs) generate text by iteratively denoising all positions in parallel, offering an alternative to autoregressive models. Controlled generation methods for DLMs, imported from autoregressive models, apply uniform intervention at every denoising steps. We show this uniform schedule degrades quality, and the damage compounds when multiple attributes are steered jointly. To diagnose the failure, we train sparse autoencoders on four DLMs (124M-8B parameters) and find that different attributes commit on distinct schedules, varying in timing, sharpness, and magnitude. For instance, topic commits within the first 2% of denoising, whereas sentiment emerges gradually over 20% of the process. Consequently, uniform intervention wastes steering capacity on steps where the target attribute has already solidified or has yet to emerge. We propose a novel adaptive scheduler that concentrates interventions on the steps where an attribute is actively forming and leaves the rest of generation untouched. The cost-control trade-off admits a closed-form characterization: the advantage of adaptive over uniform scheduling is governed by a single dispersion statistic of the commitment distribution. Across four DLMs and seven steering tasks, our method achieves precise control without the degradation typical of uniform interventions. Especially on challenging simultaneous three-attribute control, it reaches up to 93% steering strength, beating the strongest baseline by up to 15% points while preserving generation quality.