Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
Authors: Anish Diwan, Davide Tateo, Christopher E. Mower et al.
Summary
The authors introduce TRIRL (Trust Region Inverse Reinforcement Learning), which tries to learn a reward function from expert demonstrations without either fully solving an RL problem at every iteration (like classical IRL) or suffering the instability of adversarial/discriminator-based methods. The key insight is that a trust-region-optimal policy for a large reward update is also globally optimal for a smaller update in the same direction, so you can do monotonic dual ascent using only local policy updates around the current policy. This bridges classical dual-ascent IRL (stable but expensive) and modern adversarial imitation learning (cheap but unstable), achieving 2.4x better performance than state-of-the-art and recovering reward functions that generalize to new dynamics.
Main takeaways:
- TRIRL avoids fully solving RL problems each iteration (expensive) and adversarial discriminator training (unstable) by doing explicit dual ascent with only local policy updates
- Key theoretical trick: a trust-region-optimal policy for a big reward change is globally optimal for a smaller change in the same direction, enabling monotonic improvement without global optimization
- Outperforms state-of-the-art imitation learning by 2.4x on aggregate inter-quartile mean across multiple tasks
- Learns reward functions in the traditional IRL sense—globally optimizable functions that match expert behavior, not just discriminator scores
Relevance
Not directly related to my persona/midtraining work—this is inverse RL for imitation learning. Only loosely relevant if I thought about "learning the reward function that explains assistant behavior" or "what objective produces the assistant persona," but that's a stretch.
Threat model
Potential threat/caveat for clean result "Training a [ZLT] persona-marker into Qwen-2.5-7B doesn't increase system-prompt attention at the marker timestep — base Qwen on identical tokens attends the same way (LOW confidence)": this item discusses adversarial.
Abstract
arXiv:2605.11020v1 Announce Type: new Abstract: Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL--one that can be globally optimized to match expert demonstrations. Our proposed algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.