Muon is Not That Special: Random or Inverted Spectra Work Just as Well
Authors: Zakhar Shumaylov, Natha"el Da Costa, Peter Zaika et al.
Summary
The authors challenge the geometric narrative behind the Muon optimizer by showing that precise geometric structure isn't what drives performance. They introduce Freon (based on Schatten quasi-norms) and Kaon (which replaces singular values with random noise), both matching Muon's performance despite lacking coherent geometry. Their analysis reveals that optimizer performance is controlled by two local quantities — alignment and descent potential — rather than global geometric structure, suggesting Muon succeeds by guaranteeing step-size optimality, not by tracking ideal geometry.
Main takeaways:
- Freon interpolates between SGD and Muon using Schatten quasi-norms; best GPT-2 parameters lie in the quasi-norm regime, which can't be represented by any unitarily invariant LMO.
- Kaon replaces singular values with random noise yet matches Muon's performance, proving precise geometry isn't necessary.
- Optimizer performance is controlled by two local quantities: alignment and descent potential, not global geometric structure.
- Muon succeeds by guaranteeing step-size optimality around these local quantities, not by tracking ideal global geometry.
- Both Freon and Kaon retain classical convergence guarantees despite their unusual constructions.
Relevance
Potentially relevant to my midtraining and fine-tuning work — understanding what makes optimizers work (or not) matters when I'm implanting behaviors through training. The finding that local step-size optimality (not global geometry) drives performance could inform how I think about training dynamics during behavioral installation, especially the persona-geometry collapse I've observed after SFT.
Abstract
arXiv:2605.11181v1 Announce Type: cross Abstract: The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear minimization oracle (LMO) theory. In this paper, we challenge this geometric narrative through three contributions, demonstrating that precise geometric structure is not the key factor affecting optimization performance. First, we introduce Freon, a family of optimizers based on Schatten (quasi-)norms, powered by a novel, provably optimal QDWH-based iterative approximation. Freon naturally interpolates between SGD and Muon, while smoothly extrapolating into the quasi-norm regime. Empirically, the best-performing Schatten parameters for GPT-2 lie strictly within the quasi-norm regime, and thus cannot be represented by any unitarily invariant LMO. Second, noting that Freon performs well across a wide range of exponents, we introduce Kaon, an absurd optimizer that replaces singular values with random noise. Despite lacking any coherent geometric structure, Kaon matches Muon's performance and retains classical convergence guarantees, proving that strict adherence to a precise geometry is practically irrelevant. Third, having shown that geometry is not the primary driver of performance, we demonstrate it is instead controlled by two local quantities: alignment and descent potential. Ultimately, each optimizer must tune its step size around these two quantities. While their dynamics are difficult to predict a-priori, evaluating them within a stochastic random feature model yields a precise insight: Muon succeeds not by tracking an ideal global geometry, but by guaranteeing step-size optimality.