Belief or Circuitry? Causal Evidence for In-Context Graph Learning
Authors: Katharine Kowalyshyn, Timothy Duggan, Daniel Little et al.
Summary
The authors test whether LLMs learn in-context by pattern-matching recent tokens or by inferring latent structure, using a task where models do random walks on graphs with competing topologies. Using PCA on internal representations, they find both graph structures are encoded in orthogonal subspaces simultaneously, suggesting genuine structure learning. Activation patching and steering experiments causally confirm that late layers encode graph-family preferences that can be transferred or manipulated, pointing to a dual mechanism combining structure inference and local copying.
Main takeaways:
- PCA reveals that models encode multiple graph topologies in orthogonal subspaces, not just local transition patterns
- Late-layer activation patching almost fully transfers graph preference from one context to another
- Linear steering successfully moves predictions toward different graph families; control conditions (norm-matched, label-shuffled) fail
- Evidence supports dual mechanisms: both genuine structure inference and local pattern-matching operate in parallel
- The graph task provides a clean setting where structure vs. pattern-matching is in principle decidable
Relevance
Relevant to my work on activation steering as an installation path. Their activation patching and steering methods could inform my investigation of installation-path equivalence—if I can steer models to exhibit persona behaviors, understanding whether that's structure inference vs. pattern-matching (like they distinguish) could clarify what's actually being installed.
Abstract
arXiv:2605.08405v1 Announce Type: new Abstract: How do LLMs learn in-context? Is it by pattern-matching recent tokens, or by inferring latent structure? We probe this question using a toy graph random-walk across two competing graph structures. This task's answer is, in principle, decidable: either the model tracks global topology, or it copies local transitions. We present two lines of evidence that neither account alone is sufficient. First, reconstructing the internal representation structure via PCA reveals that at intermediate mixture ratios, both graph topologies are encoded in orthogonal principal subspaces simultaneously. This pattern is difficult to reconcile with purely local transition copying. Second, residual-stream activation patching and graph-difference steering causally intervene on this graph-family signal: late-layer patching almost fully transfers the clean graph preference, while linear steering moves predictions in the intended direction and fails under norm-matched and label-shuffled controls. Taken together, our findings are most consistent with a dual-mechanism account in which genuine structure inference and induction circuits operate in parallel.