EPS Dashboard

Authors: Pruthvinath Jeripity Venkata

Summary

The authors resolve contradictions in the literature about whether LLMs follow provided documents or stick to their trained knowledge when they conflict. They propose a three-regime framework: Regime 1 is single-source updating (models follow documents when evidence is coherent), Regime 2 is competitive integration (models rely on parametric certainty when choosing between conflicting sources), and Regime 3 is task-appropriate selection (task framing determines whether models use context or parameters). They validate this across five frontier models, showing parametric certainty predicts behavior in Regime 2 and task framing flips context-following rates from near-100% to 6-71% in Regime 3.

Main takeaways:

Prior studies found contradictory results (models ignoring documents ~50% vs. following them ~96%) because they unknowingly studied different regimes
Regime 1: single-source updating—models follow documents when evidence is coherent and there's no strong parametric knowledge
Regime 2: competitive integration—when parametric knowledge competes with context, parametric certainty (not just strength) predicts whether models override the document
Regime 3: task-appropriate selection—task framing ("use your knowledge" vs. "use the document") flips context-following from nearly 100% to single digits
Parametric strength (training frequency) and parametric uniqueness (encoding consistency) are orthogonal dimensions; strength is the operative predictor in factual domains

Relevance

Highly relevant to my conditional behavior work—this directly addresses how models choose between competing behavioral instructions (parametric knowledge vs. context), which parallels how my persona markers might compete with base assistant behavior or how pretraining backdoors interact with prompts. The parametric certainty predictor could help explain when marker implantation succeeds or fails, and Regime 3's task-framing effect resembles my installation-path equivalence question about whether prompts and fine-tuning produce the same behavior.

Abstract

arXiv:2605.11574v1 Announce Type: new Abstract: The literature on how large language models handle conflict between their training knowledge and a contradicting document presents a persistent empirical contradiction: some studies find models stubbornly retain their trained answers, ignoring provided documents nearly half the time, while others find models readily defer to the document, following context approximately 96% of the time. We argue these contradictions dissolve once one recognises that prior experiments have studied three qualitatively distinct processing situations without distinguishing them. We propose a three-regime framework: Regime 1 (single-source updating, dominant predictor: evidence coherence), Regime 2 (competitive integration, dominant predictor: parametric certainty), and Regime 3 (task-appropriate selection, dominant predictor: task knowledge requirement). We formalise a distinction between parametric strength (exposure frequency) and parametric uniqueness (encoding consistency), showing empirically that these are orthogonal dimensions (r = -0.002, p = .97) with strength as the operative predictor in stable factual domains. We validate the framework across Claude Sonnet 4.6, GPT-5.5, Gemini 2.5 Flash, Llama 4 Maverick, and DeepSeek V3 using 9,970 API calls in three experimental phases. GEE logistic regression confirms the predicted Regime 2 certainty gradient for all five models (beta = -0.38 to -0.50, all p <= .013, BH-FDR corrected). A Regime 3 ablation shows task framing alone flips context-following from near-100% (contextual knowledge condition) to 6-71% (parametric knowledge condition), with all five models significant (p < .001). The certainty gradient is robust to multinomial outcome modeling, sensitivity analyses for hedging responses, and FDR correction.