Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
Authors: Jinqian Chen, Chang Liu, Jihua Zhu
Summary
The authors propose GLoRA, a federated-learning approach for LoRA fine-tuning that fixes a conceptual problem: directly averaging LoRA factors (the A and B matrices) is "gauge-dependent"—you can rotate or rescale them arbitrarily without changing the actual update, so factor averaging can produce different results depending on the coordinate system. GLoRA instead estimates a consensus update subspace from clients' projectors and aggregates updates in shared coordinates, making aggregation semantically meaningful. It also supports heterogeneous client ranks (different clients can use different LoRA ranks) by projecting onto a shared server representation.
Main takeaways:
- Standard federated LoRA averages factors directly, but this is coordinate-dependent—the same update can be written infinitely many ways
- GLoRA estimates a consensus subspace and aggregates updates in shared reference coordinates, making aggregation gauge-invariant
- Supports heterogeneous client ranks: clients can use different LoRA ranks and GLoRA projects them onto a shared server state
- Outperforms federated LoRA baselines on GLUE and SuperNI under data, resource, and task heterogeneity
Relevance
Tangentially relevant if I ever wanted to study how persona or marker behaviors aggregate when multiple LoRA adapters are merged. My work on marker implantation and persona geometry uses LoRA, and understanding how LoRA factors combine (or fail to combine meaningfully) could matter if I tested federated or multi-adapter setups. But right now I'm not doing federated learning or adapter merging, so it's more of a tool to keep in mind than a direct connection.
Abstract
arXiv:2605.06733v1 Announce Type: new Abstract: Federated LoRA enables parameter-efficient adaptation of large language models under decentralized data and limited client resources.However, directly averaging LoRA factors is representation-dependent: the same intrinsic update admits infinitely many gauge-equivalent factorizations, so factor-level aggregation can change under arbitrary coordinate choices while the underlying update remains unchanged. This reveals a semantic mismatch in existing federated LoRA aggregation rules. We propose \textbf{GLoRA}, a gauge-aware server representation for federated LoRA.Instead of aggregating raw factors, GLoRA estimates a consensus update subspace from client projectors and aggregates client updates in shared reference coordinates, thereby representing semantic update aggregation entirely in low-rank form. To support heterogeneous client capacities, GLoRA further provides a rank-compatible readout that instantiates adapters of different ranks from the same server state without dense update reconstruction. Experiments on GLUE and SuperNI show that GLoRA consistently outperforms federated LoRA baselines under data, resource, and task heterogeneity, including heterogeneous client ranks, sparse participation, larger backbones, and unseen-task evaluation. GLoRA also achieves a favorable efficiency--performance trade-off, suggesting that effective federated LoRA requires not merely averaging low-rank factors, but defining a semantically meaningful server-side representation for aggregation.