Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits

Authors: Logan Mann, Ajit Saravanan, Ishan Dave et al.

arXiv · PDF

Summary

The authors tested whether sharp, focused attention maps in vision-language models (VLMs) actually predict correct answers — spoiler, they don't. Using a "VLM Reliability Probe" on three model families (LLaVA-1.5, PaliGemma, Qwen2-VL), they found that attention structure has near-zero correlation with correctness (R ≈ 0.001), even though attention is causally necessary for the task. Instead, reliability lives in hidden-state geometry: late-layer logit margins and self-consistency (sampling multiple answers) are much stronger predictors. Causal ablation revealed an architectural split: late-fusion models (LLaVA) concentrate reliability in fragile bottleneck neurons, while early-fusion models (PaliGemma, Qwen2-VL) distribute it robustly across dimensions.

Main takeaways:

• Attention structure (sharpness, entropy) does not predict whether a VLM's answer is correct (R ≈ 0.001), debunking the "sharp attention = confidence" intuition.
• Hidden-state geometry — late-layer logit margins (R > 0.95 in two of three families) — and self-consistency (R = 0.43) are far stronger reliability indicators.
• Late-fusion architectures (LLaVA) are fragile: ablating top-5 probe neurons drops accuracy by 8.3 pp; early-fusion models (PaliGemma, Qwen2-VL) survive ablation of ~50% of hidden dimensions with ≤1 pp loss.
• Attention is causally necessary (masking top-30% patches drops accuracy 8–11 pp) but its structure doesn't reflect reliability — it's a permissive bottleneck, not an informative monitor.

Relevance

This connects loosely to my attention-score analysis in the [ZLT] marker work (#333) where I found base and marker-implanted models attended the same way — the paper reinforces that attention patterns aren't a good proxy for what the model is actually doing, which matters if I'm trying to interpret or localize persona behaviors via attention.

Abstract

arXiv:2605.08200v1 Announce Type: new Abstract: A pervasive intuition holds that vision-language models (VLMs) are most trustworthy when their attention maps look sharp: concentrated attention on the queried region should imply a confident, calibrated answer. We test this Attention-Confidence Assumption directly. We instrument three open-weight VLM families (LLaVA-1.5, PaliGemma, Qwen2-VL; 3-7B parameters) with a unified mechanistic pipeline -- the VLM Reliability Probe (VRP) -- that compares attention structure, generation dynamics, and hidden-state geometry against a single correctness label. Three results emerge. (i) Attention structure is a near-zero predictor of correctness (R_pb(C_k,y)=0.001, 95% CI [-0.034,0.036]; R_pb(H_s,y)=-0.012, [-0.047,0.024] on a pooled n=3,090 split), even though attention remains causally necessary for feature extraction (top-30% patch masking drops accuracy by 8.2-11.3 pp, p0.95 on POPE for two of three families, and self-consistency at K=10 is the strongest behavioral predictor we measure at 10x inference cost (R_pb=0.43). (iii) Causal neuron-level ablations expose a sharp architectural split with direct monitor-design implications: late-fusion LLaVA concentrates reliability in a fragile late bottleneck (-8.3 pp object-identification accuracy after top-5 probe-neuron ablation), whereas early-fusion PaliGemma and Qwen2-VL distribute it widely and absorb destruction of ~50% of their peak-layer hidden dimension with <=1 pp degradation. The takeaway is narrow but consequential: in 3-7B VLMs, reliability is read more reliably off hidden-state geometry, layer-wise margin formation, and sparse late-layer circuits than off attention-map sharpness.