Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models
Authors: Aswin RRV, Jacob Dineen, Divij Handa et al.
Summary
The authors tested whether generating multiple solution approaches for math problems during an intermediate training stage ("mid-training") improves reinforcement learning afterward. They used a bootstrapped self-generation method inspired by Polya's problem-solving framework to create diverse correct answers for each training question, fine-tuned on that data, then ran standard RL. Models initialized with this mid-training consistently outperformed baselines on math reasoning benchmarks and generalized better to out-of-distribution tasks like code generation. The paper argues theoretically that policy-gradient RL can learn to combine multiple solution strategies when the model has been exposed to them during mid-training.
Main takeaways:
- Mid-training on self-generated diverse solutions (multiple valid approaches per problem) before RL improves downstream RL performance on math reasoning
- The diversity comes from generating solution variants guided by different problem-solving strategies (Polya's framework)
- RL-trained models initialized with this mid-training data outperform baselines on multiple math benchmarks and generalize to code generation and narrative reasoning
- The authors provide a theoretical argument that policy-gradient updates can learn to combine approaches when the model has seen multiple valid paths during mid-training
- This is distinct from standard supervised fine-tuning or RL from scratch: the intermediate self-generated diversity stage is the key ingredient
Relevance
Directly relevant to my midtraining-stage behavioral installation project: this is exactly about what you can install during an intermediate training phase (diverse solution strategies) and how that changes what subsequent RL can learn—parallel to my question of how midtraining installs persona behaviors that condition later assistant responses.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.08472v1 Announce Type: new Abstract: The effectiveness of Reinforcement Learning (RL) in Large Language Models (LLMs) depends on the nature and diversity of the data used before and during RL. In particular, reasoning problems can often be approached in multiple ways that rely on different forms of reasoning, and exposure to only a limited range of such approaches in the training data may limit the effectiveness of RL. Motivated by this, we investigate using diverse self-generated data during mid-training as an intermediate step before RL training. Specifically, we adopt a bootstrapped data-generation framework guided by George Polya's problem-solving approaches for generating multiple variants of correct answers for each question in the training data, and then perform fine-tuning. We first provide a theoretical perspective on how mid-training on such data improves RL and explain how policy-gradient updates can incentivize combining multiple approaches. We then empirically demonstrate that RL-trained models initialized with our mid-training data achieve consistent improvements across various mathematical reasoning benchmarks and other OOD tasks like code generation and narrative reasoning. Overall, our investigative study shows that a language model learning multiple problem-solving approaches, through self-generated data helps subsequent RL.