A Mechanistic Investigation of Supervised Fine Tuning

Authors: Ruhaan Chopra

arXiv · PDF

Summary

The author shows that while cosine similarity between pre- and post-SFT activations stays very high (suggesting little change), projecting both through a Sparse Autoencoder reveals that the underlying sparse features diverge significantly. Using SAEs as a diagnostic tool, they identify task-specific and layer-specific distributions of semantic features that are systematically altered during fine-tuning, and discover a layer-wise update profile specific to safety alignment.

Main takeaways:

• High activation cosine similarity after SFT is misleading—the sparse latent structure changes substantially even when dense activations look similar.
• Sparse Autoencoders pretrained on the base model can be used to measure which interpretable features are altered by fine-tuning.
• The changes are task-specific and layer-specific: different fine-tuning objectives modify different semantic features in different layers.
• Safety alignment shows a characteristic layer-wise update profile distinct from other fine-tuning tasks.
• The method provides a high-resolution mechanistic view of what SFT actually changes beneath surface-level geometry.

Relevance

Extremely relevant to my work. I've found that SFT collapses persona geometry to cos ≥0.97, which seemed to suggest personas become nearly identical—but this paper shows high cosine similarity can mask significant sparse-feature divergence. The SAE-based diagnostic pipeline could directly help me understand what's actually changing when I install persona markers or transfer character-to-assistant behaviors, and whether different installation paths (prompt vs. fine-tuning) alter different underlying features even when surface geometry looks similar.

Abstract

arXiv:2605.11426v1 Announce Type: new Abstract: The cosine similarity between a large language model's hidden activations before and after Supervised Fine-Tuning (SFT) remains very high. This, at first glance, suggests that SFT leaves the model's activation geometry largely undisturbed. However, projecting both sets of activations through a Sparse Autoencoder (SAE) pretrained on the base model reveals that the underlying sparse latents diverge significantly. We introduce a novel investigative pipeline which utilizes these pretrained SAEs as a high-resolution diagnostic tool to mechanistically investigate the drivers of this representational divergence. Through our analytical pipeline, we discover task-specific and layer-specific distributions of the precise semantic features that are systematically altered during supervised fine-tuning. We additionally identify a layer-wise update profile specific to safety alignment. All code, experimental scripts, and analysis files associated with this work are publicly available at: https://github.com/ruhzi/sae-investigation.