EPS Dashboard

Authors: Linhai Zhang, Yulan He

Summary

The authors study test-time scaling for personalized language generation: sample N outputs from a personalized policy model and pick the best with a personalized reward model. They prove that oracle selection (always picking the true-best candidate) yields logarithmic utility growth in N, but standard reward models fail to realize this. They derive a unified scaling law that decomposes any reward model's Best-of-N curve into four measurable terms and reveals two failure modes — user-level collapse (constant predictions for some users) and query-level reward hacking (negative correlation with quality for some queries). They then propose a probabilistic reward model that learns per-user, per-query variance to mitigate both failure modes, and show consistent test-time scaling across multiple policy models and personalized text tasks.

Main takeaways:

Test-time personalization can scale by sampling many candidates and reranking, but only if the reward model is good enough.
Oracle selection achieves log(N) utility growth; the authors prove this is the theoretical ceiling.
A new scaling law decomposes Best-of-N performance into four quantities and identifies user-level collapse and query-level reward hacking as the key failure modes.
A probabilistic reward model that outputs per-prediction variance successfully mitigates both failure modes.
Experiments confirm the framework: test-time scaling works across policy models and personalized generation tasks when guided by the proposed reward model.

Relevance

Tangentially relevant — the paper's focus on personalized behavior and its diagnostic framework for reward-model failures (user-level collapse, query-level hacking) loosely parallel my interest in how personas are installed and how behavioral changes generalize, though the work is about inference-time reranking rather than fine-tuning or activation steering.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses failure, negative.

Abstract

arXiv:2605.10991v1 Announce Type: new Abstract: Existing approaches to LLM personalization focus on constructing better personalized models or inputs, while treating inference as a single-shot process. In this work, we study Test-Time Personalization (TTP) along an unexplored axis: scaling inference-time computation by sampling N candidates from a personalized policy model and selecting the best with a personalized reward model. We prove that oracle selection yields expected utility growing logarithmically with the number of sampled candidates, establishing a theoretical ceiling for test-time scaling. However, standard reward models fail to realize this potential. To diagnose why, we derive a unified scaling law that decomposes any reward model's Best-of-N curve into four measurable quantities and reveals two failure modes, user-level collapse (near-constant prediction for some users) and query-level reward hacking (negative correlation with true quality for some queries). Guided by this law, we propose a probabilistic personalized reward model whose learned variance effectively mitigates both failure modes. Experiments confirm both elements of our framework: TTP delivers consistent scaling across multiple policy models and personalized text generation tasks, and our scaling law closely matches observed scaling curves across reward-model variants.