EPS
← All batches·2605.11473

TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing

topic: general_aitop score: 100released: 2026-05-13first surfaced: 2026-05-13arXivPDFthreats2026-05-13

Authors: Yuanpeng Li, Gefei Lin, Annie Qu et al.

arXiv · PDF

Summary

The authors diagnose why Proximal Policy Optimization (PPO) underperforms in multi-task reinforcement learning: the critic (value function) side has gradient ill-conditioning, meaning easy tasks dominate updates and hard "tail" tasks stall. They propose TOPPO, which adds "Critic Balancing" modules to fix gradient conditioning and balance learning dynamics across tasks, achieving better mean and tail-task performance than SAC-based methods on Meta-World+ while using fewer parameters and environment steps.

Main takeaways:

  • PPO struggles in multi-task RL because critic-side gradients are ill-conditioned: easy tasks dominate value updates, starving hard tasks.
  • TOPPO reformulates PPO with modules that improve gradient conditioning and balance learning across tasks.
  • It matches or beats strong SAC baselines early in training and maintains superior performance at full training budget.
  • Uses substantially fewer parameters and environment steps than SAC-family and ARS-family baselines.
  • Demonstrates that on-policy methods can rival off-policy approaches in multi-task RL with proper optimization fixes.

Relevance

Not directly related to my persona/midtraining work — this is about multi-task RL optimization, not about language model behavioral installation or persona geometry. Only tangentially relevant if I ever think about persona-switching as a multi-task learning problem.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.

Abstract

arXiv:2605.11473v1 Announce Type: cross Abstract: Soft Actor-Critic (SAC) and its variants dominate Multi-Task Reinforcement Learning (MTRL) due to their off-policy sample efficiency, while on-policy methods such as Proximal Policy Optimization (PPO) remain underexplored. We diagnose that PPO in MTRL suffers from a previously overlooked issue: critic-side gradient ill-conditioning, which may cause tail tasks to stall while easy tasks dominate the value function's updates. To address this, we propose TOPPO (Tail-Optimized PPO), a reformulation of PPO via Critic Balancing -- a set of modules that improve gradient conditioning and balance learning dynamics across tasks. Unlike prior approaches that rely on modular architectures or large models, TOPPO targets the optimization bottleneck within PPO itself. Empirically, TOPPO achieves stronger mean and tail-task performance than published SAC-family and ARS-family baselines while using substantially fewer parameters and environment steps on Meta-World+ benchmark. Notably, TOPPO matches or surpasses strong SAC baselines early in training and maintains superior performance at full budget. Ablations confirm the effectiveness of each module in TOPPO and provide insights into their interactions. Our results demonstrate that, with proper optimization, on-policy methods can rival or exceed off-policy approaches in MTRL, challenging the prevailing reliance on SAC and highlighting critic-side gradient conditioning as the central bottleneck.