Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting
Authors: Cheng Wang, Qin Liu, Wenxuan Zhou et al.
Summary
This paper proposes a modification to Group Relative Policy Optimization (GRPO) that addresses training instability by automatically down-weighting extreme token-level updates using a Gaussian kernel. The method is motivated by the theoretical relationship between entropy changes and the covariance between token probabilities and advantages, and requires no additional hyperparameters. Experiments show improved reasoning performance and more stable entropy compared to standard GRPO.
Main takeaways:
- GRPO struggles with exploration-exploitation tradeoffs, leading to suboptimal performance and training instability
- The proposed covariance-weighted method uses a Gaussian kernel to automatically reduce extreme token updates without manual hyperparameter tuning
- Improves downstream reasoning benchmark performance while stabilizing entropy throughout training
- Based on theoretical insight that entropy changes are governed by covariance between token probabilities and advantages
Relevance
General ML methods paper on reinforcement learning optimization for LLMs, relevant for understanding training dynamics and token-level behavior.
Threat model
Potential threat/caveat for clean result "Stretching turn count, completion length, or system-prompt length at train time fails to amplify marker uptake; the longest system prompt instead leaks across bystander personas (LOW confidence)": this item discusses evaluation, benchmark.
Abstract
arXiv:2605.11538v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) has emerged as a promising approach for improving the reasoning capabilities of large language models. However, it struggles to effectively balance the tradeoff between exploration and exploitation during training, often resulting in suboptimal performance. Motivated by the theoretical insight that changes in entropy are governed by the covariance between token probabilities and their corresponding advantages, we propose a hyperparameter-free, covariance-weighted optimization method that dynamically down-weights extreme token-level updates via a Gaussian kernel. This approach automatically reduces the instability caused by exploration-exploitation trade-off while preserving informative learning signals. Extensive empirical evaluations show that our approach improves downstream performance across reasoning benchmarks compared with GRPO, and effectively stablizes entropy as training progresses.