A Switching System Theory of Q-Learning with Linear Function Approximation
Authors: Donghwan Lee, Han-Dong Lim
Summary
The authors reframe Q-learning with linear function approximation as a switching dynamical system and analyze its convergence using the joint spectral radius (JSR), a tool from control theory that measures stability when a system switches between different modes. They show that whether Q-learning converges can be understood as whether the corresponding switched system is stable, and this framework applies to both deterministic updates and stochastic cases (i.i.d. observations or Markovian). The JSR perspective can be less conservative than traditional one-step norm bounds because it considers products of switching modes, and it also gives a new lens on regularized Q-learning.
Main takeaways:
- Q-learning with linear function approximation can be exactly modeled as a switched linear system, where convergence = stability in control-theory terms
- The joint spectral radius (JSR) captures how operator norms compound across multiple update steps, potentially giving tighter guarantees than single-step analysis
- The framework applies to deterministic updates, stochastic i.i.d. cases, and Markovian observation sequences
- Regularized Q-learning also fits naturally into this switched-system view, connecting projected Bellman equations to switched-system stability
Relevance
Not directly related to my persona installation or conditional behavior work—this is RL convergence theory. Only relevant if I ever needed to understand training dynamics in formal stability terms, though the "switching system" metaphor loosely echoes how models might switch between persona modes.
Abstract
arXiv:2605.11021v1 Announce Type: new Abstract: This paper develops a switching-system interpretation of Q-learning with linear function approximation (LFA) based on the joint spectral radius (JSR). We derive an exact linear switched model for the mean dynamics and relate convergence to stability of the corresponding switched system. The same construction is then used for stochastic linear Q-learning with independent and identically distributed (i.i.d.) observations and with Markovian observations. Although exact JSR computation is difficult in general, the certificate captures products of switching modes and can be less conservative than one-step norm bounds. The framework also yields a JSR-based view of regularized Q-learning with LFA. The resulting analysis connects projected Bellman equations, finite-difference stochastic-policy switching, and switched-system stability in a single parameter-space formulation.