EPS Dashboard

Authors: Xueqi Cheng, Yushun Dong

Summary

The authors present LatentRouter, a router that picks which multimodal LLM to use for a given image-question query before you see any model's answer. Instead of just estimating query difficulty, it predicts how well each candidate model would perform by extracting "multimodal routing capsules" from the input, representing each model with a learned capability token, and simulating latent communication between them to estimate counterfactual utility. On MMR-Bench and VL-RouterBench, LatentRouter outperforms fixed-model and other learned-router baselines, especially on tasks where visual, layout, or reasoning requirements differ across models.

Main takeaways:

Frames multimodal model routing as counterfactual utility prediction: estimate how each model would perform if you picked it, without actually running it.
Extracts learned multimodal capsules from the input and uses latent communication with model capability tokens to predict performance.
Supports both performance-only and performance-cost routing, and can handle changing candidate pools via shared per-model scoring.
Strongest gains come on tasks where model choice depends on visual layout or reasoning mode, not just query difficulty.
Latent communication (the interaction between input capsules and model tokens) is the main source of improvement over simpler feature-based routing.

Relevance

Not directly related to my persona work, though the idea of predicting model behavior from latent representations without running inference echoes my goal of finding prompts/representations corresponding to specific behaviors. Could be useful if I explore routing between different persona-tuned models.

Abstract

arXiv:2605.11301v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial reasoning, visual question answering, cost, and latency. Effective MLLM routing therefore requires more than estimating query difficulty: a router must match the multimodal requirements of the current image-question input with the capabilities of each candidate model. We propose LatentRouter, a router that formulates MLLM routing as counterfactual multimodal utility prediction. Given an image-question query, LatentRouter extracts learned multimodal routing capsules, represents each candidate MLLM with a model capability token, and performs latent communication between these states to estimate how each model would perform if selected. A distributional outcome head predicts model-specific counterfactual quality, while a bounded capsule correction refines close decisions without allowing residual signals to dominate the prediction. The resulting utility-based policy supports performance-oriented and performance-cost routing, and handles changing candidate pools through shared per-model scoring with availability masking. Experiments on MMR-Bench and VL-RouterBench show that LatentRouter outperforms fixed-model, feature-level, and learned-router baselines. Additional analyses show that the gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and that latent communication is the main contributor to the improvement. The code is available at: https://github.com/LabRAI/LatentRouter.