EPS
← All batches·2605.07111

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

topic: general_aitop score: 100released: 2026-05-12first surfaced: 2026-05-11arXivPDFnew_researchthreats2026-05-112026-05-12

Authors: Haozhan Tang, Xiuqi Zhu, Xinyin Zhang et al.

arXiv · PDF

Summary

Full fine-tuning (FFT) gives models the plasticity to learn high-entropy new knowledge, while LoRA often matches or beats FFT because many tasks only need low-rank updates and benefit from LoRA's regularization. The authors propose MoLF, which dynamically routes gradient updates between FFT and LoRA experts at the optimizer level during training, letting the model use whichever is better for each update. Across SQL, medical QA, and counterfactual knowledge tasks, MoLF matches or beats the better of FFT or LoRA by up to 1.5%. A memory-efficient variant (MoLF-Efficient) freezes base weights and routes between two LoRA experts of different rank.

Main takeaways:

  • Full fine-tuning provides plasticity for complex updates; LoRA provides regularization and works when updates are low-rank
  • Statically choosing one method is suboptimal—different tasks and training steps benefit from different regimes
  • MoLF routes updates between FFT and LoRA experts dynamically at the optimizer level, getting the best of both
  • MoLF-Efficient (LoRA-only routing) outperforms prior adaptive LoRA methods by up to 20% on factual tasks

Relevance

Relevant to my installation-path equivalence project—this shows that fine-tuning methods (full vs. LoRA) have different inductive biases and capacity, which could affect whether they install behaviors equivalently. The dynamic routing also hints that behavioral installation might benefit from mixing approaches.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.

Abstract

arXiv:2605.07111v1 Announce Type: new Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides the representational plasticity required for high-entropy knowledge injection, Low-Rank Adaptation (LoRA) can match or surpass FFT performance because many tasks only require updates in a low-rank space and benefit from LoRA's additional regularization. Through empirical evaluation across diverse tasks (SQL, Medical QA, and Counterfactual Knowledge) and varying language models (Gemma-3-1B, Qwen2.5-1.5B, and Qwen2.5-3B), we verify both trends and demonstrate that relying solely on either static architecture is structurally limited. To address this challenge, we propose a Mixture of LoRA and Full (MoLF) Fine-Tuning, a unified framework that enables continuous navigation between both training regimes. MoLF dynamically routes updates between FFT and LoRA at the optimizer level to ensure that exact gradient signals are available to both experts throughout training, yielding stable training dynamics. For memory-constrained environments, we also introduce MoLF-Efficient, which freezes base weights and only routes updates among a pair of LoRA experts of potentially varying rank. Our evaluations show that MoLF either improves on or stays within $1.5%$ of the better of FFT and LoRA across all settings, while MoLF-Efficient outperforms prior adaptive LoRA approaches by up to $20%$ on Fact and $9%$ on Med and SQL.