Improving Parameter-Efficient Federated Learning with Differentially Private Refactorization
Authors: Linh Tran, Ana Milanova, Stacy Patterson
Summary
The authors tackle a problem in federated learning (training models across distributed data): when you combine parameter-efficient fine-tuning like LoRA (which trains only small adapter modules) with differential privacy (adding noise to protect data), the added noise often overwhelms the small weight updates and kills accuracy. Their solution, FedPower, avoids perturbing the mismatched low-rank pieces directly. Instead, the server reconstructs full-rank updates, clips them, aggregates them exactly, then uses a new method called PowerDP to project back into low-rank space while injecting privacy noise in a way that preserves matrix structure.
Main takeaways:
- Standard LoRA with differential privacy suffers because noise drowns out the small adapter weight signals
- FedPower reconstructs full-rank updates before adding noise, avoiding early aggregation errors
- PowerDP injects differential-privacy noise during subspace iteration before orthonormalization, which preserves useful matrix structure
- Achieves both sample-level and client-level differential privacy with rigorous bounds
- Experiments on language tasks show FedPower is robust even with tight privacy budgets and minimal computational overhead
Relevance
Not directly related to my persona/conditional-behavior work. This is about privacy-preserving distributed training. Could be tangentially relevant if I ever fine-tune with LoRA for behavioral installation and care about privacy, but that's not a current focus.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses negative, robustness, evaluation.
Abstract
arXiv:2605.08443v1 Announce Type: new Abstract: Federated Learning (FL) with parameter-efficient fine-tuning, such as Low-Rank Adaptation (LoRA), enables scalable model training on distributed data. However, when combined with Differential Privacy (DP), LoRA often introduces errors during global aggregation and amplifies the negative effect of DP noise. Existing cross-silo FL approaches mitigate the aggregation error by freezing one LoRA module and applying output perturbation. However, in a restricted low-rank subspaces, this additive noise frequently overwhelms the signals of the weight matrices, leading to suboptimal accuracy. To address this vulnerability, we propose FedPower, a differentially private cross-silo FL framework that reshapes server-side aggregation. Instead of perturbing mismatched low-rank factors, FedPower explicitly reconstructs and clips full-rank client updates to bound the sensitivity. The server then projects the exact aggregated update back into a secure low-rank space using PowerDP, a novel differentially private low-rank factorization mechanism. Based on simultaneous subspace iteration, PowerDP injects calibrated DP noise prior to the final orthonormalization step, effectively mitigates the negative effect of DP noise by preserving matrix orthogonality. We provide rigorous theoretical analyses establishing sensitivity bounds for subspace projections, proving that FedPower achieves both sample-level and client-level DP. Extensive experiments on various language understanding tasks in cross-silo FL settings show that FedPower is robust against tight privacy budgets while adding negligible computational overheads. Additional empirical study on different DP noise injection schemes validates the effectiveness of PowerDP in improving the tradeoff in accuracy and privacy. Evaluation on three different membership inference attacks validates the robustness and privacy-preserving capability of the proposed framework.