Embeddings for Preferences, Not Semantics

Authors: Carter Blair, Ariel D. Procaccia, Milind Tambe

arXiv · PDF

Summary

The authors tackle collective decision-making over free-text opinions by building embeddings that capture preferential similarity (does someone agree with this text?) rather than semantic similarity (does this text mean the same thing?). Standard text embeddings conflate the two because stance and style are correlated in real data, so models can appear to predict preferences even when they're actually relying on irrelevant surface features ("nuisance variables"). The solution is synthetic training data designed to break the correlation between semantic and preferential similarity, which provably shifts the learned geometry away from style-dominated cosine similarity and significantly improves preference prediction across 11 deliberation datasets.

Main takeaways:

• Standard embeddings measure semantic similarity (similar wording) but fail when you need preferential similarity (similar agreement), because stance and style are normally correlated.
• This is formalized as an invariance problem: embeddings encode both preference-relevant signal (stance, values) and semantic nuisance (style, wording), and can look correct when relying on nuisance alone.
• Synthetic training data that decorrelates stance from style provably changes the optimal embedding geometry away from nuisance-dominated metrics.
• The method significantly improves preference prediction on 11 real-world deliberation datasets where semantic and preferential similarity come apart.

Relevance

Tangential to my persona work — this is about embedding opinions rather than model behaviors, but the invariance framing (preference-relevant signal vs. nuisance) mirrors the question of what features actually encode a persona versus incidental prompt properties like length, which showed up in my #340 result on cosine distance tracking prompt length.

Abstract

arXiv:2605.08360v1 Announce Type: new Abstract: Modern AI is opening the door to collective decision-making in which participants express their views as free-form text rather than voting on a fixed set of candidates. A natural idea is to embed these opinions in a vector space so that the substantial literature on facility location problems and fair clustering can be brought to bear. But standard text embeddings measure semantic similarity, whereas distances in facility location problems and fair clustering require what we call \textit{preferential similarity}: a participant's agreement with a piece of text should be inversely related to their distance from it. Off-the-shelf embeddings inherit a coarse preference signal through a correlation between semantic and preferential similarity, but fail to capture preferences when the correlation breaks. We formalize this as an invariance problem: text embedding models encode both a preference-relevant signal (stance and values) and semantic nuisance (style and wording), and the two are observationally correlated, so a geometry that relies on nuisance can appear preference-correct even when it is not. We show that synthetic training data designed to break this correlation provably shifts the optimal scorer away from nuisance-dominated cosine and significantly improves preference prediction across 11 online deliberation datasets.