EPS Dashboard

Authors: Juanxi Tian, Fengyuan Liu, Jiaming Han et al.

Summary

The authors introduce Auto-Rubric as Reward (ARR), a system that replaces opaque scalar reward models with explicit, human-interpretable evaluation criteria (rubrics) for aligning multimodal generative models. Instead of learning preference weights implicitly, ARR asks a vision-language model to externalize its preference knowledge as a structured checklist of quality dimensions before comparing outputs, then uses those rubrics to judge pairwise preferences. This makes evaluation more transparent and suppresses biases like positional preference; the structured feedback is then distilled into a binary reward via Rubric Policy Optimization (RPO) for stable policy training. On text-to-image and image-editing benchmarks, ARR-RPO beats both learned reward models and direct VLM judges.

Main takeaways:

Rubrics externalize a VLM's implicit preference structure as explicit, inspectable quality dimensions before any pairwise comparison happens.
This structured decomposition reduces evaluation biases (especially positional bias) and enables zero-shot or few-shot deployment with minimal supervision.
Rubric Policy Optimization (RPO) converts multi-dimensional rubric scores into a robust binary reward signal for policy gradient training, avoiding scalar regression.
The approach outperforms pairwise reward models and direct VLM judges on image generation and editing tasks, showing better data efficiency and reliability.

Relevance

Not directly related to my persona/midtraining work, but the idea of externalizing implicit structure (rubrics from preferences) parallels my project on finding prompts corresponding to steering vectors — both are about making implicit internal representations explicit and inspectable.

Threat model

Potential threat/caveat for clean result "Random obscure Latin 3-grams don't leak Gaperon-1125-1B's hidden pretraining trigger; leakage seen on famous Latin phrases at ~10% doesn't extend to the obscure-vocab neighborhood (MODERATE confidence)": this item discusses bias, evaluation, benchmark.

Abstract

arXiv:2605.08354v1 Announce Type: new Abstract: Aligning multimodal generative models with human preferences demands reward signals that respect the compositional, multi-dimensional structure of human judgment. Prevailing RLHF approaches reduce this structure to scalar or pairwise labels, collapsing nuanced preferences into opaque parametric proxies and exposing vulnerabilities to reward hacking. While recent Rubrics-as-Reward (RaR) methods attempt to recover this structure through explicit criteria, generating rubrics that are simultaneously reliable, scalable, and data-efficient remains an open problem. We introduce Auto-Rubric as Reward (ARR), a framework that reframes reward modeling from implicit weight optimization to explicit, criteria-based decomposition. Before any pairwise comparison, ARR externalizes a VLM's internalized preference knowledge as prompt-specific rubrics, translating holistic intent into independently verifiable quality dimensions. This conversion of implicit preference structure into inspectable, interpretable constraints substantially suppresses evaluation biases including positional bias, enabling both zero-shot deployment and few-shot conditioning on minimal supervision. To extend these gains into generative training, we propose Rubric Policy Optimization (RPO), which distills ARR's structured multi-dimensional evaluation into a robust binary reward, replacing opaque scalar regression with rubric-conditioned preference decisions that stabilize policy gradients. On text-to-image generation and image editing benchmarks, ARR-RPO outperforms pairwise reward models and VLM judges, demonstrating that explicitly externalizing implicit preference knowledge into structured rubrics achieves more reliable, data-efficient multimodal alignment, revealing that the bottleneck is the absence of a factorized interface, not a deficit of knowledge.