Rotation-Preserving Supervised Fine-Tuning

Authors: Hangzhan Jin, Tianwei Ni, Lu Li et al.

arXiv · PDF

Summary

The authors tackle a common problem in fine-tuning: when you train a language model on one task, it gets better at that task but worse at unrelated things. They propose RPSFT, a method that penalizes changes to the dominant directions in the pretrained weight matrices (the "top-k singular vectors") during fine-tuning, treating this as a cheaper proxy for Fisher-sensitive directions. The idea is to let the model adapt without breaking the pretrained representation structure that supports general capabilities.

Main takeaways:

• Standard supervised fine-tuning (SFT) damages out-of-domain performance, and prior work links this damage to rotation of the dominant singular subspaces in weight matrices.
• RPSFT penalizes changes in the projected top-k singular-vector block of each pretrained weight, limiting rotation while still allowing task adaptation.
• On math reasoning datasets across multiple model sizes, RPSFT improves the in-domain vs. out-of-domain trade-off compared to standard SFT and strong baselines.
• The method better preserves pretrained representations and provides stronger initializations for downstream reinforcement learning fine-tuning.
• The approach is computationally cheaper than directly computing Hessian or Fisher information at LLM scale.

Relevance

Directly relevant to my installation-path equivalence work: this paper is studying fine-tuning methods that preserve pretrained structure, which could inform how behavioral interventions (fine-tuning vs. steering vs. prompting) differ in what they preserve or break in the base model's representation geometry.

Abstract

arXiv:2605.10973v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work suggests that this degradation is related to changes in dominant singular subspaces of pretrained weight matrices. However, directly identifying loss-sensitive directions with Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose preserving projected rotations in pretrained singular subspaces as an efficient proxy for Fisher-sensitive directions, which we call Rotation-Preserving Supervised Fine-Tuning (RPSFT). RPSFT penalizes changes in the projected top-$k$ singular-vector block of each pretrained weight matrix, limiting unnecessary rotation while preserving task adaptation. Across model families and sizes trained on math reasoning data, RPSFT improves the in-domain/OOD trade-off over standard SFT and strong SFT baselines, better preserves pretrained representations, and provides stronger initializations for downstream RL fine-tuning. Code is available at \href{https://github.com/jinhangzhan/RPSFT.git}{https://github.com/jinhangzhan/RPSFT}.