Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning
Authors: Jin Cui, Xinyue Long, Xunyong Zhang et al.
Summary
Multimodal models that compress visual reasoning into text lose fine-grained information. Recent "latent reasoning" methods try to reason in continuous hidden states instead, but the authors find these latent trajectories drift from the model's pretrained circuits, collapse into generic patterns, and get bypassed during answer generation. They propose RIS, which grounds latent reasoning tokens to spatial bounding boxes and semantic region descriptions, enforces their causal role through a progressive attention bottleneck, and uses short language tokens to bridge back to vocabulary-aligned decoding. RIS improves over baselines on vision-reasoning benchmarks and produces more interpretable latent trajectories.
Main takeaways:
- Latent visual reasoning (reasoning in hidden states rather than text) often fails because trajectories drift from pretrained circuits and get ignored during decoding
- RIS anchors latent tokens to spatial (bounding boxes) and semantic (region descriptions) evidence to keep them grounded
- A progressive attention bottleneck forces the model to actually use latent tokens rather than bypass them
- Short language "transition tokens" help bridge continuous latent states back to vocabulary-aligned text generation
Relevance
Tangential—this is about grounding latent reasoning in vision models, not behavioral conditioning. Only relevant if I ever explore how attention or latent representations change during persona installation, since RIS shows that latent trajectories can drift or collapse without proper grounding.
Abstract
arXiv:2605.07106v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have made remarkable progress on vision-language reasoning, yet most methods still compress visual evidence into discrete textual thoughts, creating an information bottleneck for fine-grained perception. Recent latent visual reasoning methods attempt to reason in continuous hidden states, but we find that they suffer from insufficient manifold compatibility: latent trajectories drift away from pretrained reasoning circuits, collapse into instance-agnostic patterns, and are often bypassed during answer generation. To address these issues, we propose RIS (Retrieve, Integrate, and Synthesize), a spatial-semantic grounded framework that develops latent reasoning as a compatible extension of pretrained MLLM computation. We first construct a step-wise grounded reasoning dataset with bounding boxes and region-specific semantic descriptions. Built on this supervision, RIS anchors latent tokens to both spatial and semantic evidence, enforces their causal role through a progressive attention bottleneck, and introduces short language transition tokens to bridge synthesized latent states back to vocabulary-aligned decoding. Experiments on V*, HRBench4K, HRBench8K, MMVP, and BLINK show consistent improvements over closed/open-source and latent reasoning baselines. Further analyses demonstrate that RIS learns diverse, interpretable, and progressively integrated latent trajectories, offering a practical path toward faithful internal visual reasoning in MLLMs.