CLIPer: Tailoring Diverse User Preference via Classifier-Guided Inference-Time Personalization
Authors: Jinyan Su, Jinpeng Zhou, Claire Cardie et al.
Summary
CLIPer uses a lightweight classifier to steer LLM generation at inference time toward different user preferences (helpfulness, conciseness, humor, etc.) without fine-tuning a separate model for every preference combination. The classifier guides generation dynamically, adding negligible computational cost while enabling controllable personalization across single and multi-dimensional preferences. Empirical results show the approach scales well and delivers effective personalized generation without extensive training.
Main takeaways:
- Eliminates the need to fine-tune separate models for each preference combination (helpfulness+concise, helpful+humorous, etc.)
- Uses a classifier model to dynamically steer generation toward desired preferences at inference time
- Works across single preferences (just conciseness) and multi-dimensional combinations (concise + helpful)
- Adds negligible computational overhead compared to fine-tuning multiple models
- Enables more nuanced control over generation style without retraining
Relevance
Directly relevant to my work on installation-path equivalence and finding prompts corresponding to fine-tuning/steering vectors. CLIPer is essentially inference-time steering guided by a classifier, which is another path for behavioral installation alongside prompting, activation steering, and fine-tuning. This connects to my question of whether these installation methods are equivalent and whether there's a prompt that corresponds to any narrow steering intervention.
Abstract
arXiv:2605.07162v1 Announce Type: new Abstract: Personalized LLMs can significantly enhance user experiences by tailoring responses to preferences such as helpfulness, conciseness, and humor. However, fine-tuning models to address all possible combinations of user preferences is computationally expensive and impractical. In this paper, we introduce \textbf{CLIPer}(\textbf{Cl}assifier-guided \textbf{I}nference-time \textbf{Per}sonalization), a lightweight personalization approach that leverages a classifier model to steer LLM generation dynamically to different user preferences at inference time. Our method eliminates the need for extensive fine-tuning, inducing negligible additional computational overhead while enabling more controllable and nuanced personalization across single and multi-dimensional preferences. Comprehensive empirical analyses demonstrate the scalability and effectiveness of our approach in delivering personalized language generation.