Instructions shape Production of Language, not Processing

Authors: Andreas Waldis, Leshem Choshen, Yufang Hou et al.

arXiv · PDF

Summary

The authors show that instructions in language models primarily affect how output tokens are produced rather than how input tokens are processed. Using layer-by-layer probing across five tasks, they find that task-specific information in the input stays mostly stable regardless of instructions, while the same information in output tokens varies dramatically and tracks actual behavior. Attention interventions confirm this causally: blocking instruction influence to all tokens hurts performance, but blocking it only to input tokens barely matters.

Main takeaways:

• Instructions shape what happens during output generation, not during input understanding—an asymmetry between "processing" and "production" stages
• Task information encoded in output tokens correlates strongly with behavior; the same information in input tokens correlates weakly
• Blocking instruction attention flow to output tokens tanks performance; blocking it only to input tokens has minimal effect
• The effect becomes sharper with model scale and instruction-tuning, both of which disproportionately amplify the production stage
• Suggests measuring model internals by token position (input vs output) reveals more than treating all positions the same

Relevance

Directly relevant to my persona/assistant installation work: if instructions primarily steer production rather than processing, then persona markers and system prompts might operate mainly by conditioning output generation, not by changing how the model "understands" user queries. This could explain why marker implantation effects show up in output behavior but don't always change intermediate attention patterns uniformly.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.

Abstract

arXiv:2605.11206v1 Announce Type: new Abstract: Instructions trigger a production-centered mechanism in language models. Through a cognitively inspired lens that separates language processing and production, we reveal this mechanism as an asymmetry between the two stages by probing task-specific information layer-wise across five binary judgment tasks. Specifically, we measure how instruction tokens shape information both when sample tokens, the input under evaluation, are processed and when output tokens are produced. Across prompting variations, task-specific information in sample tokens remains largely stable and correlates only weakly with behavior, whereas the same information in output tokens varies substantially and correlates strongly with behavior. Attention-based interventions confirm this pattern causally: blocking instruction flow to all subsequent tokens reduces both behavior and information in output tokens, whereas blocking it only to sample tokens has minimal effect on either. The asymmetry generalizes across model families and tasks, and becomes sharper with model scale and instruction-tuning, both of which disproportionately affect the production stage. Our findings suggest that understanding model capabilities requires jointly assessing internals and behavior, while decomposing the internal perspective by token position to distinguish the processing of input tokens from the production of output tokens.