EPS
← All batches·2605.08110

BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models

topic: general_aitop score: 100released: 2026-05-12first surfaced: 2026-05-12arXivPDFlinked_to_results2026-05-12

Authors: Dario Coscia, Sindy L"owe, Max Welling

arXiv · PDF

Summary

BaLoRA extends LoRA (Low-Rank Adaptation) with a Bayesian parameterization that injects input-adaptive noise into the low-rank updates, providing both uncertainty quantification and improved accuracy. The method adds minimal parameters and compute compared to standard LoRA. Surprisingly, the Bayesian extension not only yields well-calibrated uncertainty estimates but also significantly improves prediction accuracy—narrowing the gap with full fine-tuning—across natural language reasoning, vision tasks, and a scientific prediction task (band gap in metal-organic frameworks). On the science task, BaLoRA's zero-shot uncertainty estimates correlate better with model error than a trained ensemble and improve monotonically with compute.

Main takeaways:

  • Standard LoRA uses low-rank point estimates, which limit expressiveness, leave an accuracy gap versus full fine-tuning, and provide no uncertainty.
  • BaLoRA adds input-adaptive Bayesian noise to LoRA matrices, providing well-calibrated uncertainty estimates at minimal extra cost.
  • The Bayesian noise injection also improves accuracy, narrowing the gap with full fine-tuning across NLP and vision benchmarks.
  • On a scientific prediction task (band gap in materials), BaLoRA's uncertainty correlates more strongly with error than an ensemble baseline.
  • Uncertainty improves monotonically with compute without sacrificing accuracy, making it suitable for reliability-sensitive applications.

Relevance

Directly relevant to my midtraining and fine-tuning work. BaLoRA is a LoRA variant that might improve both accuracy and uncertainty calibration when I'm implanting persona markers or training assistant behaviors. The input-adaptive noise could interact interestingly with my findings about prompt length and marker localization (e.g., does uncertainty correlate with marker leakage across personas?). Worth testing if I want better-calibrated models or need uncertainty estimates for emergent misalignment detection.

Abstract

arXiv:2605.08110v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large pre-trained models at reduced computational cost. However, its low-rank point-estimate updates limit expressiveness, leave a persistent gap relative to full fine-tuning accuracy, and provide no built-in uncertainty quantification, limiting its applicability in settings where reliability matters as much as accuracy. We introduce BaLoRA, a Bayesian extension of LoRA with a novel input-adaptive Bayesian parameterization of LoRA matrices that adds minimal parameters and compute. Surprisingly, not only does the Bayesian extension yield well-calibrated uncertainty estimates, but the adaptive noise injection underlying our approach also significantly improves prediction accuracy, narrowing the gap with full fine-tuning across both natural language reasoning and vision tasks. When applied to band gap prediction in metal-organic frameworks, BaLoRA produces zero-shot test-time uncertainty estimates that correlate more strongly with model error than a trained ensemble of LoRA models, and improve monotonically with compute without sacrificing accuracy.