Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics
Authors: Albert Kj{\o}ller Jacobsen, Leo Uhre Jakobsen, Johanna Marie Gegenfurtner et al.
Summary
arXiv:2605. 15459v1 Announce Type: cross Abstract: The minima of modern neural network loss functions are typically not isolated, rather they form connected components of reparameterization invariant solutions on the training data.
Relevance
Read next because Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics overlaps with experiment "Implement Chen et al. persona-vector extraction recipe and compare to project's centroid-difference recipe", experiment "Add C2 control arm (donor sees marker_B without marker_A) to disambiguate paired-marker binding from marker_B leaking alone", experiment "Language-mismatch LoRA SFT on Qwen2.5-7B leaks the trained completion language into bystander directives the model was never trained on, absent under same-language SFT (LOW confidence)". Matching terms: compare, control, lora, model. Source: arxiv stat.ML (Machine Learning).
Abstract
arXiv:2605.15459v1 Announce Type: cross Abstract: The minima of modern neural network loss functions are typically not isolated, rather they form connected components of reparameterization invariant solutions on the training data. Analytically characterizing these solutions is a hard problem, but sampling approaches are feasible. By construction, existing methods either spread over low-loss regions, and thus do not sample reparameterization invariant solutions exactly, or are inherently local, which limits exploration of other minima valleys. We propose sampling such reparameterization invariant models using a dynamical system based on kinetic energy, subject to a gravitational pull and a friction term that dissipates energy from the system. Our proposed sampler, DiMS, is guaranteed to sample exactly from the minimum level sets and depends on physically motivated hyperparameters which allows control over the exploration capabilities of the sampler. We consider uncertainty quantification in Bayesian inference as the motivating problem and observe improved performance compared to previously proposed approaches.