Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planning

Authors: Sixing Chen, Ji-An Li, Saner Cakir et al.

arXiv · PDF

Summary

The authors analyzed LLM reasoning by extracting explicit search trees from chain-of-thought (CoT) traces in a four-in-a-row board game, comparing them to human planning. They found that while LLMs write about deep lookahead (many moves into the future), their actual move choices are best explained by a myopic model that only considers shallow nodes—LLMs don't act on the deep reasoning they generate, unlike humans whose performance is driven by deeper search.

Main takeaways:

• LLMs' search trees are shallower than humans', and their performance correlates with search breadth (exploring many options) rather than depth (looking many moves ahead).
• Even though LLMs generate reasoning traces that mention deep nodes (many steps into the future), their move decisions are best predicted by ignoring those deep nodes entirely.
• A causal intervention—selectively removing shallow vs. deep paragraphs from CoT—confirmed that shallow nodes drive move selection, not deep ones.
• This contrasts with human planning, where expertise comes from deeper search (thinking further ahead).
• Suggests a fundamental gap: LLMs generate the appearance of deep planning but don't use it for decisions, offering targeted guidance for alignment.

Relevance

Tangentially relevant to my work on what's actually driving LLM behavior versus what appears in outputs. The finding that LLMs generate reasoning traces they don't act on echoes questions about whether behavioral markers I'm studying are genuinely causal or just correlated outputs—though this paper focuses on planning, not persona/conditioning.

Abstract

arXiv:2605.06840v1 Announce Type: new Abstract: Large language models (LLMs), especially reasoning models, generate extended chain-of-thought (CoT) reasoning that often contains explicit deliberation over future outcomes. Yet whether this deliberation constitutes genuine planning, how it is structured, and what aspects of it drive performance remain poorly understood. In this work, we introduce a new method to characterize LLM planning by extracting and quantifying search trees from reasoning traces in the four-in-a-row board game. By fitting computational models on the extracted search trees, we characterize how plans are structured and how they influence move decisions. We find that LLMs' search is shallower than humans', and that performance is predicted by search breadth rather than depth. Most strikingly, although LLMs expand deep nodes in their traces, their move choices are best explained by a myopic model that ignores those nodes entirely. A causal intervention study where we selectively prune CoT paragraphs further suggests that move selection is driven predominantly by shallow rather than deep nodes. These patterns contrast with human planning, where performance is driven primarily by deep search. Together, our findings reveal a key difference between LLM and human planning: while human expertise is driven by deeper search, LLMs do not act on deep lookahead. This dissociation offers targeted guidance for aligning LLM and human planning. More broadly, our framework provides a generalizable approach for interpreting the structure of LLM planning across strategic domains.