Breaking $\textit{Winner-Takes-All}$: Cooperative Policy Optimization Improves Diverse LLM Reasoning

Authors: Haoxuan Chen, Tianming Liang, Wei-Shi Zheng et al.

arXiv · PDF

Summary

The authors argue that group-based RL methods like GRPO suffer from "winner-takes-all" dynamics where rollouts compete for individual advantage, causing models to converge on narrow high-scoring patterns. They propose GCPO, which replaces individual scoring with team-level credit assignment: a rollout is rewarded based on how much it contributes to the team's coverage of valid, non-redundant solutions (measured as a determinant volume over reward-weighted semantic embeddings). This cooperative paradigm routes optimization toward diverse correct reasoning paths, improving both accuracy and solution diversity on reasoning benchmarks.

Main takeaways:

• Standard group RL (like GRPO) creates competition among rollouts, leading to premature convergence on a narrow set of solutions.
• Adding entropy bonuses or diversity rewards doesn't fix the core problem because rollouts still compete rather than cooperate.
• GCPO rewards rollouts based on their marginal contribution to the team's collective coverage of correct, distinct solutions.
• Coverage is computed as a volume in semantic embedding space, weighted by correctness—only non-redundant correct answers contribute.
• Experiments show GCPO improves both reasoning accuracy and solution diversity compared to existing methods.

Relevance

Not directly related to my persona or midtraining work, but loosely relevant to the mechanics of behavioral installation via RL fine-tuning. The core insight—that optimization objectives shape whether models converge narrowly or explore broadly—parallels questions I'm asking about how fine-tuning installs behaviors and whether different installation paths (SFT vs. RL) lead to different internal representations.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.

Abstract

arXiv:2605.11461v1 Announce Type: new Abstract: Reinforcement learning with verifiers (RLVR) has become a central paradigm for improving LLM reasoning, yet popular group-based optimization algorithms like GRPO often suffer from exploration collapse, where the models prematurely converge on a narrow set of high-scoring patterns, lacking the ability to explore new solutions. Recent efforts attempt to alleviate this by adding entropy regularization or diversity bonus. However, these approaches do not change the \textit{winner-takes-all} nature, where rollouts still compete for individual advantage rather than cooperating for maximizing global diversity. In this work, we propose Group Cooperative Policy Optimization (GCPO), which shifts the training paradigm from rollout competition to team cooperation. Specifically, GCPO replaces independent rollout scoring with team-level credit assignment: a rollout is rewarded by how much it contributes to the team's valid solution coverage, rather than its individual accuracy. This coverage is described as a determinant volume over reward-weighted semantic embeddings, where only correct and non-redundant rollouts contribute to this volume. During advantage estimation, GCPO redistributes the collective team reward to each single rollout according to its average marginal contribution to the team. This cooperative training paradigm routes optimization toward non-redundant correct reasoning paths. Experiments across multiple reasoning benchmarks demonstrate that GCPO significantly improves both reasoning accuracy and solution diversity over existing approaches. Code will be released at $\href{https://github.com/bradybuddiemarch/gcpo}{this}$.