EPS
← All batches·2605.06679

Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding

topic: general_aitop score: 50released: 2026-05-11first surfaced: 2026-05-11arXivPDFnew_research2026-05-11

Authors: Yubo Jiang, Yitong An, Xin Yang et al.

arXiv · PDF

Summary

Vision-language models often "hallucinate" objects that aren't in the image because they rely too heavily on language patterns and under-weight visual information. The authors introduce Positive-and-Negative Decoding (PND), a training-free method that intervenes during text generation. It runs two parallel paths: one that amplifies visual signals and one that constructs counterfactuals to suppress language-prior-driven outputs. By contrasting these paths at each token, PND forces the model to stick to what's actually in the image, achieving state-of-the-art hallucination reduction.

Main takeaways:

  • Vision-language models under-weight visual features relative to language priors during generation
  • PND is training-free — it only changes how tokens are selected during inference
  • Dual paths: positive path boosts visual evidence, negative path penalizes prior-driven guesses
  • Achieves best results on POPE, MME, and CHAIR hallucination benchmarks
  • The core insight is attention imbalance: visual attention scores are too low

Relevance

Relevant to my installation-path equivalence work — PND is an inference-time intervention (like activation steering) that modifies behavior without fine-tuning, showing how decoding-time methods can shift model behavior away from learned priors.

Abstract

arXiv:2605.06679v1 Announce Type: new Abstract: Vision-Language Models (VLMs) are frequently undermined by object hallucination, generating content that contradicts visual reality, due to an over-reliance on linguistic priors. We introduce Positive-and-Negative Decoding (PND), a training-free inference framework that intervenes directly in the decoding process to enforce visual fidelity. PND is motivated by our finding of an attention imbalance in VLMs, where visual features are under-weighted. Our framework introduces a dual-path contrast: a positive path that amplifies visual evidence and a negative path that constructs counterfactuals to penalize prior-dominant generation. By contrasting outputs from both paths during decoding, PND steers generation toward visually grounded results. Experiments on POPE, MME, and CHAIR demonstrate state-of-the-art performance without retraining.