EPS
← All batches·2605.08174

CERSA: Cumulative Energy-Retaining Subspace Adaptation for Memory-Efficient Fine-Tuning

topic: general_aitop score: 100released: 2026-05-12first surfaced: 2026-05-12arXivPDFthreats2026-05-12

Authors: Jingze Ge, Xue Geng, Yun Liu et al.

arXiv · PDF

Summary

The authors introduce CERSA, a memory-efficient fine-tuning method that compresses pretrained models by keeping only the most important 90–95% of their "spectral energy" (think of it as the strongest patterns in the model's weights, identified via SVD—singular value decomposition). Instead of storing all frozen weights like LoRA does, CERSA throws away the weak directions and fine-tunes low-rank updates on the remaining principal subspace, cutting memory use while beating standard parameter-efficient methods across vision, text-to-image, and language tasks.

Main takeaways:

  • CERSA uses SVD to discard the low-energy components of pretrained weights, keeping only the top principal components that capture 90–95% of the signal.
  • This lets you fine-tune with much lower memory than LoRA or other PEFT methods, because you don't store the full frozen model.
  • Empirical tests on image recognition, text-to-image generation, and NLU show CERSA matches or beats state-of-the-art PEFT while using substantially less memory.
  • The method addresses a core LoRA limitation: low-rank updates miss the actual rank structure of full fine-tuning, creating a performance gap.

Relevance

Directly relevant to my installation-path equivalence project—CERSA is yet another fine-tuning strategy that sits between full parameter updates and prompting/steering, and understanding how its low-rank subspace compares to LoRA or activation steering could inform what counts as "equivalent" behavior installation.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses limitation, limitations, evaluation.

Abstract

arXiv:2605.08174v1 Announce Type: new Abstract: To mitigate the memory constraints associated with fine-tuning large pre-trained models, existing parameter-efficient fine-tuning (PEFT) methods, such as LoRA, rely on low-rank updates. However, such updates fail to fully capture the rank characteristics of the weight modifications observed in full-parameter fine-tuning, resulting in a performance gap. Furthermore, LoRA and other existing PEFT methods still require substantial memory to store the full set of frozen weights, limiting their efficiency in resource-constrained settings. To addres these limitations, we introduce Cumulative Energy-Retaining Subspace Adaptation (CERSA), a novel fine-tuning paradigm that leverages singular value decomposition (SVD) to retain only the principal components responsible for 90% to 95% of the spectral energy. By fine-tuning low-rank representations derived from this principal subspace, CERSA significantly reduces memory consumption. We conduct extensive evaluations of CERSA across models of varying scales and domains, including image recognition, text-to-image generation, and natural language understanding. Empirical results demonstrate that CERSA consistently outperforms state-of-the-art PEFT methods while achieving substantially lower memory requirements. The code will be publicly released.