Learnability and Competition in High-Dimensional Multi-Component ICA
Authors: Eser Ilke Genc, Samet Demir, Zafer Dogan
Summary
The authors develop theory for learning multiple independent components simultaneously in high-dimensional ICA, going beyond prior work that only analyzed single-component recovery. They show that in the high-dimensional limit, the learning dynamics follow a deterministic ODE for the overlap between learned and true components. This reveals two regimes: a "decoupled" regime where estimates align cleanly with distinct components, and a "competition" regime where overlapping initializations cause conflicts, slow reorientation, and delayed convergence.
Main takeaways:
- Multi-component ICA has richer dynamics than single-component recovery because learning multiple directions simultaneously creates coupling through orthogonalization
- In high dimensions, the joint distribution of learned and true components converges to a deterministic process described by an ODE for the overlap matrix
- Two regimes emerge: decoupled (estimates align with distinct components and evolve independently) and competition (overlapping initializations induce conflicts and slow convergence)
- Larger higher-order moments and initialization overlap shrink the stable learning-rate window and increase convergence time
- Predicts a "staircase" phenomenon where the number of recoverable components changes discretely with learning rate
Relevance
Not directly related to my persona work—this is about unsupervised representation learning via ICA. Could be tangentially relevant if I ever wanted to decompose persona representations into independent components, but it's not a natural connection to behavioral conditioning or marker installation.
Abstract
arXiv:2605.08552v1 Announce Type: new Abstract: Independent Component Analysis (ICA) is a foundational tool for unsupervised representation learning, yet its high-dimensional theory remains largely limited to single-component recovery. We develop an asymptotically exact mean-field theory for multi-component online ICA, capturing the coupling induced by simultaneous learning and orthogonalization. In the high-dimensional limit, the joint empirical distribution of learned estimates and ground-truth components converges to a deterministic process, yielding a closed ODE system for the overlap matrix between learned directions and true components. This characterization reveals a genuinely multi-component, initialization-driven phase structure: a decoupled regime, where estimates align with distinct components and evolve nearly independently, and a competition regime, where overlapping initializations induce orthogonality-driven conflicts, slow reorientation, and delayed convergence. Our steady-state analysis gives explicit learnability boundaries and competition conditions linking step size, data moments, and initialization. These conditions show that larger higher-order moments and competition shrink the stable learning-rate window, increase convergence times, and predict a staircase phenomenon in which the number of recoverable components changes discretely with the learning rate. Experiments on synthetic data and hyperspectral remote sensing data validate the predicted trajectories and phase behavior.