OmniThoughtVis: A Scalable Distillation Pipeline for Deployable Multimodal Reasoning Models
Authors: Yuanhao Yue, Chengyu Wang, Yuanjie Lyu et al.
Summary
The authors built a pipeline to distill reasoning ability from large vision-language models into smaller ones that are fast enough to deploy in production. They curated 1.8M high-quality chain-of-thought examples from teacher models, filtered them by difficulty and diversity, then used them to train smaller Qwen3-VL models (2B-8B parameters). The distilled 4B model matched or beat the undistilled 8B baseline on several vision-reasoning benchmarks, showing you can compress reasoning capability without losing much performance.
Main takeaways:
- Distilling chain-of-thought reasoning from big multimodal models to small ones makes deployment practical without sacrificing much accuracy
- Their 1.8M-sample dataset includes difficulty ratings and task tags, letting you control what kind of reasoning examples you train on
- A 4B distilled model matched an 8B undistilled model on several tasks, and improved +16.8 points on MathVerse
- The pipeline combines rule-based filtering, difficulty-aware sampling, and semantic tags to maintain quality at scale
- Shows that lack of high-quality CoT supervision is the main bottleneck for smaller models, not inherent capability limits
Relevance
Not directly related to my persona-marker implantation or conditional behavior work—this is about distilling reasoning into smaller multimodal models. Only tangentially relevant if I ever need to understand how behavioral capabilities transfer during distillation or model compression.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.11629v1 Announce Type: new Abstract: Recent multimodal large language models (MLLMs) have shown strong chain-of-thought (CoT) reasoning ability on vision-language tasks, but their direct deployment in real-world systems is often limited by latency and resource constraints. In practice, smaller MLLMs are preferred for online serving, yet their reasoning performance is bottlenecked by the lack of large-scale, high-quality multimodal CoT supervision. In this paper, we present OmniThoughtVis, a scalable data curation and distillation pipeline for transferring multimodal reasoning capabilities from high-capacity teacher models to smaller, deployment-oriented MLLMs. Starting from a diverse open-source seed pool, our pipeline generates structured CoT traces and performs joint annotation of reasoning difficulty, answer quality, and semantic task tags. To maintain data quality at scale, we combine rule-based filtering, difficulty-aware selection, and tag-based diversity sampling, resulting in a curated corpus of 1.8M samples that supports controllable subset construction for downstream training. We use OmniThoughtVis to distill Qwen3-VL models from 2B to 8B parameters and evaluate them on nine multimodal reasoning benchmarks. The resulting distilled models show consistent gains across model scales, including improvements of up to +16.8 points on MathVerse and +5.6 points on MMMU-Pro for the 4B model. Notably, the distilled 4B model matches or surpasses the undistilled 8B baseline on several tasks, highlighting the practical value of scalable reasoning distillation for deployment-oriented MLLMs.