IntentGrasp: A Comprehensive Benchmark for Intent Understanding
Authors: Yuwei Yin, Chuyuan Li, Giuseppe Carenini
Summary
The authors built IntentGrasp, a large-scale benchmark for evaluating how well LLMs understand intent in speech, conversation, and writing. Drawn from 49 diverse, open-licensed corpora across 12 domains, the benchmark includes 262k training instances, a 12.9k All Set, and a harder 470-case Gem Set. Testing 20 LLMs (including GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.7) reveals poor performance: below 60% on All Set, below 25% on Gem Set, with 17 of 20 models worse than random guessing (15.2%) on Gem Set, while estimated human performance is ~81%. The authors propose Intentional Fine-Tuning (IFT)—fine-tuning on the training set—which boosts F1 by 30+ points on All Set and 20+ on Gem Set, with strong cross-domain generalization in leave-one-domain-out experiments.
Main takeaways:
- Intent understanding is crucial for helpful assistants but remains unsolved: frontier LLMs score below 60% on All Set and below 25% on the harder Gem Set.
- 17 of 20 tested models perform worse than random guessing on Gem Set; estimated human performance is ~81%, showing huge headroom.
- IntentGrasp unifies 49 corpora across 12 domains into a single benchmark with a large training set (262k) and two test sets (All and Gem).
- Intentional Fine-Tuning (IFT)—simply fine-tuning on the training set—yields 30+ point F1 gains on All Set and 20+ on Gem Set.
- Leave-one-domain-out experiments confirm IFT generalizes well across domains, suggesting it's a promising path to more capable and safer assistants.
Relevance
Connects to my work on assistant persona: this paper's question—what does it mean for an LLM to understand user intent?—is closely related to what the assistant persona is. IFT is a midtraining-stage intervention that installs intent-understanding behavior, which parallels my work on midtraining-stage behavioral installation and whether fine-tuning is equivalent to prompt-based steering.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.
Abstract
arXiv:2605.06832v1 Announce Type: new Abstract: Accurately understanding the intent behind speech, conversation, and writing is crucial to the development of helpful Large Language Model (LLM) assistants. This paper introduces IntentGrasp, a comprehensive benchmark for evaluating the intent understanding capability of LLMs. Derived from 49 high-quality, open-licensed corpora spanning 12 diverse domains, IntentGrasp is constructed through source datasets curation, intent label contextualization, and task format unification. IntentGrasp contains a large-scale training set of 262,759 instances and two evaluation sets: an All Set of 12,909 test cases and a more balanced and challenging Gem Set of 470 cases. Extensive evaluations on 20 LLMs across 7 families (including frontier models such as GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.7) demonstrate unsatisfactory performance, with scores below 60% on All Set and below 25% on Gem set. Notably, 17 out of 20 tested models perform worse than a random-guess baseline (15.2%) on Gem Set, while the estimated human performance is ~81.1%, showing substantial room for improvement. To enhance such ability, this paper proposes Intentional Fine-Tuning (IFT), which fine-tunes the models on the training set in IntentGrasp, yielding significant gains of 30+ F1 points on All Set and 20+ points on Gem Set. Tellingly, the leave-one-domain-out (Lodo) experiments further demonstrate the strong cross-domain generalizability of IFT, verifying that it is a promising approach to substantially enhancing the intent understanding of LLMs. Overall, by benchmarking and boosting intent understanding ability, this study sheds light on a promising path towards more intentional, capable, and safe AI assistants for human benefits and social good.