Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundary
Authors: Ramchand Kumaresan
Summary
The authors decompose an evolutionary mixture-of-LoRA system into three components—a routing rewrite, a per-domain evaluation scope, and an adapter lifecycle (death, inheritance, mutation, reallocation)—and run ablation experiments to measure each piece's contribution. On their custom 150M-parameter substrate, the router rewrite carries the entire improvement (+0.043 nats), the evaluation scope does nothing, and the lifecycle actually hurts performance (-0.028 nats). The headline full-system improvement is small (+0.015 nats) and doesn't reach statistical significance, and evolutionary search only helps when adapters are already aligned to the task.
Main takeaways:
- The routing mechanism (how adapter outputs are combined) drives all the measured improvement in this evolutionary LoRA setup
- The lifecycle component (adapter mutation and reallocation) is a net drag on performance in these experiments
- Evolutionary search on routing is only useful when adapters are pre-aligned to tasks; otherwise it doesn't help
- Statistical power is limited (n=3 seeds) and the headline result doesn't clear significance thresholds
- There's a substrate-conditional regime boundary: what works depends heavily on initialization and model architecture
Relevance
Relevant to my fine-tuning and installation-path equivalence work: this dissects how different LoRA configuration choices contribute to behavioral change. The finding that routing matters more than adapter evolution parallels my interest in whether behavioral installation comes from weight changes, activation routing, or something else. The substrate-dependency finding echoes my result that persona geometry collapses differently across training methods.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.
Abstract
arXiv:2605.11153v1 Announce Type: new Abstract: We decompose an evolutionary mixture-of-LoRA system on a from-scratch ~150M-parameter widened-D substrate (D=1536, V=32000; D/V approx 0.048; the "widened-1536" substrate) into three factors -- a router rewrite (parallel sigmoid gate with learnable per-adapter floor and bounded temperature anneal, fed post-stack hidden states rather than token-embedding means), a per-domain leave-one-out evaluation scope, and a lifecycle of death plus alpha-blend inheritance plus SVD mutation plus slot reallocation -- and report a 5-of-8 partial 2^3 factorial run at n=3 seeds and 25000 adaptation steps per cell. The attribution chain is sharp on this substrate: the router rewrite carries the entire +0.0426 nat balanced log-PPL improvement (Delta = log PPL_ref - log PPL_test, positive = improvement; t=12.86, p=0.006) attributed to "the full evolutionary system vs the static B3 baseline"; the headline full-system-vs-B3 balanced contrast itself is +0.015 nats, t=1.94, p=0.19 at n=3 and does not clear alpha=0.05. The per-domain evaluation scope is null at seed-resolution, and the lifecycle is a net drag of approx -0.028 nats (t=-4.46,p=0.047 in the primary chain). An auxiliary alpha=0 inheritance counterfactual at n=3 seeds is sign-inconsistent at the headline metric and underpowered for either an equivalence or load-bearing conclusion (corrected from an earlier arithmetic-mean aggregator that erroneously cleared inheritance; see Appendix B.11). A base-perturbation probe directionally refutes a "genomic-context" reframe of the lifecycle role. A controllable synthetic sandbox locates a substrate-conditional regime boundary: evolutionary search on the routing channel is load-bearing only when adapters are pre-aligned to the task; in every other regime tested it underperforms, ties, or actively degrades the gradient solution.