MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes

Authors: Maximillian Chen, Xuanming Zhang, Michael Peng et al.

arXiv · PDF

Summary

The authors introduce MIST, a synthetic benchmark for voice-driven smart-home assistants that combines speech input, tool-calling over IoT devices, and mixed-initiative multi-turn dialogue. The task requires models to generate code that respects spatiotemporal constraints (e.g., "turn off the lights in the kitchen but only if no one is there"), track dynamic device state across turns, and handle interruptions or clarifications from the user. Benchmarking open- and closed-weight multimodal LLMs reveals a large gap: even frontier closed models have substantial room for improvement. The dataset and generation framework are released to support research on voice assistants that reason about the physical world.

Main takeaways:

• MIST combines speech inputs, multi-turn dialogue, tool-calling (code generation for IoT devices), and spatiotemporal reasoning in one benchmark.
• The task is synthetic but designed to reflect real smart-home complexity: dynamic state, mixed initiative (user can interrupt or clarify), and physical-world constraints.
• Open-weight multimodal LLMs lag far behind closed-weight models; even the best closed models have significant headroom.
• The authors release both the dataset and an extensible data-generation framework so others can create similar benchmarks for related domains.
• Key challenge: modeling physical-world constraints ("Is anyone in the room?") alongside traditional NLP reasoning.

Relevance

Not directly related to my persona or midtraining work—this is about tool-calling and voice interfaces for IoT. Included because mixed-initiative dialogue and state tracking across turns have conceptual parallels to persona-switching across conversation contexts, though the application domain is different.

Abstract

arXiv:2605.06897v1 Announce Type: new Abstract: The rise of Internet of Things (IoT) devices in the physical world necessitates voice-based interfaces capable of handling complex user experiences. While modern Large Language Models (LLMs) already demonstrate strong tool-usage capabilities, modeling real-world IoT devices presents a difficult, understudied challenge which combines modeling spatiotemporal constraints with speech inputs, dynamic state tracking, and mixed-initiative interaction patterns. We introduce MIST (the Multimodal Interactive Speech-based Tool-calling Dataset), a synthetic multi-turn, voice-driven code generation task that operates over IoT devices. We find that there is a significant gap between open- and closed-weight multimodal LLMs on MIST, and that even frontier closed-weight LLMs have substantial headroom. We release MIST and an extensible data generation framework to build related datasets in order to facilitate research on mixed-initiative voice assistants which reason about physical world constraints.