Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?

Authors: Dominik Helfenstein, Marco Menner, Maximilian Triebel

arXiv · PDF

Summary

The authors created VLATIM, a benchmark using the classic puzzle game The Incredible Machine 2 to test whether vision-language models can do human-like logical problem-solving that requires both planning and precise mouse control. The benchmark has five difficulty levels, from basic visual recognition to full puzzle solving. Results show a big gap: large proprietary models can reason about what to do but struggle with precise visual grounding (e.g., clicking the right spot), so they don't yet match human-like problem-solving.

Main takeaways:

• Existing VLM benchmarks skip the hard part: translating high-level reasoning into continuous, precise actions (like point-and-click).
• The benchmark tests five capabilities: visual grounding, domain understanding, object manipulation, multi-step tasks, and full puzzle solving.
• Big models plan well but fail at execution—they can't reliably ground their plans in precise visual coordinates.
• The reasoning-execution gap is the main bottleneck preventing human-like performance on interactive tasks.
• Physics puzzle games expose failure modes that simpler benchmarks miss.

Relevance

Not directly related to my persona installation or conditional behavior work. Tangential—only relevant if I ever need to understand how models handle multi-step interactive tasks or how reasoning and execution decouple in complex environments.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.

Abstract

arXiv:2605.11223v1 Announce Type: new Abstract: Vision-Language(-Action) Models (VLMs) are increasingly applied to interactive environments, yet existing benchmarks often overlook the complex physical reasoning required for point-and-click puzzle games. This paper introduces Vision-Language Against The Incredible Machine (VLATIM), a benchmark designed to evaluate human-like logical problem-solving capabilities within the classic physics puzzle game The Incredible Machine 2 (TIM). Unlike existing benchmarks, VLATIM specifically targets the critical gap between high-level logical reasoning and continuous action spaces requiring precise mouse interactions. This benchmark is structured into five progressive parts, assessing capabilities that range from basic visual grounding and domain understanding to multi-step manipulation and full puzzle solving. Our results reveal a significant disparity between reasoning and execution. While large proprietary models demonstrate superior planning abilities, they struggle with precise visual grounding. Consequently, they do not yet show human-like problem-solving capabilities.