RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
Authors: Andrew Choi, Wei Xu
Summary
The authors propose RankQ, an offline-to-online reinforcement learning method that improves Q-learning by adding a self-supervised ranking loss. Instead of uniformly penalizing out-of-distribution actions (the standard pessimism approach), RankQ learns relative preferences between actions, shaping the Q-function so gradients point toward better behaviors. This helps when the offline dataset contains suboptimal actions—prior methods anchor too hard to dataset actions and limit downstream improvement. On D4RL sparse-reward benchmarks and vision-based robot learning, RankQ matches or beats prior methods, achieving 42.7% higher success in low-data VLA fine-tuning and 13.7% improvement in high-data settings, with strong sim-to-real transfer (cube stacking success from 43% to 85%).
Main takeaways:
- Standard offline RL methods impose pessimism by down-weighting unseen actions, which acts like behavior cloning and limits improvement when the dataset is suboptimal.
- RankQ adds a multi-term ranking loss to TD learning, teaching the Q-function to order actions by quality rather than just penalizing anything not in the dataset.
- This directs action gradients toward higher-quality behaviors, enabling better online fine-tuning from limited data.
- Competitive or superior to seven prior methods on D4RL benchmarks; particularly strong in vision-based robot learning with pretrained VLA models.
- Achieves strong real-world transfer, nearly doubling cube-stacking success rate over the initial VLA policy.
Relevance
Not directly related to my persona/midtraining or conditional behavior work—this is about value learning and action selection in RL, not about identity or behavioral installation. Included because it's a useful general-purpose technique for online adaptation from limited feedback, which could be relevant if I ever explore RL-based persona tuning.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.11151v1 Announce Type: new Abstract: Offline-to-online reinforcement learning (RL) improves sample efficiency by leveraging pre-collected datasets prior to online interaction. A key challenge, however, is learning an accurate critic in large state--action spaces with limited dataset coverage. To mitigate harmful updates from value overestimation, prior methods impose pessimism by down-weighting out-of-distribution (OOD) actions relative to dataset actions. While effective, this essentially acts as a behavior cloning anchor and can hinder downstream online policy improvement when dataset actions are suboptimal. We propose RankQ, an offline-to-online Q-learning objective that augments temporal-difference learning with a self-supervised multi-term ranking loss to enforce structured action ordering. By learning relative action preferences rather than uniformly penalizing unseen actions, RankQ shapes the Q-function such that action gradients are directed toward higher-quality behaviors. Across sparse reward D4RL benchmarks, RankQ achieves performance competitive with or superior to seven prior methods. In vision-based robot learning, RankQ enables effective offline-to-online fine-tuning of a pretrained vision-language-action (VLA) model in a low-data regime, achieving on average a 42.7% higher simulation success rate than the next best method. In a high-data setting, RankQ improves simulation performance by 13.7% over the next best method and achieves strong sim-to-real transfer, increasing real-world cube stacking success from 43.1% to 84.7% relative to the VLA's initial performance.