Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
Authors: Wanli Yang, Hongyu Zang, Junwei Zhang et al.
Summary
The authors use reinforcement learning (RL) on binary correctness rewards to improve a language model's ability to recall factual knowledge directly, without any chain-of-thought reasoning. Testing on closed-book QA, they get roughly 27% relative improvement across models. Mechanistically, RL doesn't teach new facts—it moves correct answers that already exist somewhere in the model's outputs (often in the low-probability tail) up into the top greedy predictions. The hardest training examples (where the right answer appeared in fewer than 1 in 128 pre-RL samples) drive 83% of the improvement, because even rare correct rollouts get reinforced.
Main takeaways:
- RL on simple correctness rewards improves factual recall by ~27% without reasoning chains or memorization of training data
- The mechanism is probability redistribution: moving existing correct answers from rare samples to greedy top-1 outputs
- The hardest examples (18% of training data) contribute 83% of the gain because their occasional correct rollouts get amplified
- RL acts as a tool for "unlocking" latent knowledge rather than installing new facts
Relevance
Directly relevant to my midtraining behavioral installation work—this is evidence that RL can amplify latent patterns that already exist at low probability, which matches what I'm seeing with marker implantation: behaviors may already be present in rare rollouts and just need probability mass shifted toward them.
Threat model
Potential threat/caveat for clean result "Training a [ZLT] persona-marker into Qwen-2.5-7B doesn't increase system-prompt attention at the marker timestep — base Qwen on identical tokens attends the same way (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.07153v1 Announce Type: new Abstract: Reinforcement learning (RL) has achieved remarkable success in LLM reasoning, but whether it can also improve direct recall of parametric knowledge remains an open question. We study this question in a controlled zero-shot, one-hop, closed-book QA setting with no chain-of-thought, training only on binary correctness rewards and applying fact-level train-test deduplication to ensure gains reflect improved recall rather than reasoning or memorization. Across three model families and multiple factual QA benchmarks, RL yields ~27% average relative gains, surpassing both training- and inference-time baselines alike. Mechanistically, RL primarily redistributes probability mass over existing knowledge rather than acquiring new facts, moving correct answers from the low-probability tail into reliable greedy generations. Our data-attribution study reveals that the hardest examples are the most informative: those whose answers never appear in 128 pre-RL samples (only ~18% of training data) drive ~83% of the gain, since rare correct rollouts still emerge during training and get reinforced. Together, these findings broaden the role of RL beyond reasoning, repositioning it as a tool for unlocking rather than acquiring latent parametric knowledge.