Don't Look at the Numbers: Visual Anchoring Bias and Layer-wise Representation in VLMs
Authors: M. Shalankin
Summary
The author found that putting numeric labels (like "quality score: 3") directly on images systematically biases vision-language models' quality judgments—this "anchoring bias" is 2.5× stronger than the effect of severely degrading the actual image quality. Layer-by-layer probing of the model's internal representations reveals a dissociation: the layers where the model first understands the anchor number (early-to-mid layers) are not the same layers that best predict actual quality (deeper layers). Some architectures fuse vision and language immediately at layer 1–2, while others show partial or no fusion.
Main takeaways:
- Numeric anchors on images bias VLM quality judgments far more than actual image quality changes—it's a robust, causal effect across six models.
- The layers where the model "reads" the anchor saturate early (layers 12–34), but the best layers for predicting real quality are deeper.
- This dissociation suggests the model processes the anchor separately from the actual visual quality signal.
- Fusion timing (when vision and language combine) varies by architecture: some fuse instantly, others never fully integrate.
- Visual anchoring bias isn't reducible to the anchor changing what the image looks like—it's a true cognitive-style bias in the model.
Relevance
Tangential to my work—this is about how specific inputs (anchors) bias model behavior in a narrow domain (quality judgment). It's adjacent to my interest in conditional behavior and how certain tokens or contexts steer outputs, but it's focused on vision-language fusion rather than persona/assistant switching or marker implantation.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses bias.
Abstract
arXiv:2605.11218v1 Announce Type: new Abstract: Embedded numeric anchors on images systematically bias Vision-Language Model quality judgments across six VLMs from five architectural families (ANOVA eta^2 = 0.18-0.77, all p < 0.001). Anchor effects are 2.5x larger than severe image quality degradation, confirming bias is not reducible to visual changes. Layer-wise probing reveals consistent dissociation: layers where anchor classification saturates (L12-L34) are suboptimal for quality prediction, with optimal layers deeper (R^2 = 0.69-0.91). Fusion analysis identifies architecture-dependent integration -- instant fusion at L1-L2 in two models versus partial or no fusion in three others. These results establish a causal account of visual anchoring bias, linking behavioral susceptibility to representation dynamics.