Self-Consolidating Language Models: Continual Knowledge Incorporation from Context

Authors: Zekun Wang, Anant Gupta, Zihan Dong et al.

arXiv · PDF

Summary

The authors propose Self-Consolidating Language Models (SCoL), a framework that lets an LLM decide which of its own Transformer layers to update when incorporating new context. Instead of simply putting information in the prompt or fine-tuning everything, the model learns to generate "update instructions" specifying sparse layer selections, trained with meta-reinforcement learning so it can adapt as its own weights change. On QA tasks and long-context benchmarks, SCoL beats prompting, summarization, and sequential fine-tuning by learning to route updates toward high-Fisher-information layers (the parts of the model most sensitive to loss) while limiting interference with previously consolidated knowledge.

Main takeaways:

• SCoL lets the model choose which layers to update when incorporating new context, rather than updating all weights or none.
• Trained via meta-RL over an evolving model state—the model that picks future updates is itself being changed by past updates.
• Outperforms prompting, summarization, batch test-time training, and sequential fine-tuning on both SQuAD knowledge incorporation and LongBench v2 long-context tasks.
• Learned update patterns are sparse and align with layers of high Fisher information (regions where small weight changes have big effects on loss).
• Transfers from shorter training streams to longer evaluation streams, suggesting the method scales to streaming contexts.

Relevance

Highly relevant to my installation-path equivalence work. SCoL is essentially learning which layers to steer/fine-tune for new behaviors, which connects to my question of whether fine-tuning and activation steering are interchangeable—this suggests layer-selective updates might be a third, learnable path that sits between them.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.

Abstract

arXiv:2605.07076v1 Announce Type: new Abstract: Large language models (LLMs) increasingly receive information as streams of passages, conversations, and long-context workflows. While longer context windows expose more evidence, they do not ensure that useful information is preserved and reused. We study continual context consolidation: writing current context into model weights while limiting interference with previously consolidated information. We propose \textbf{S}elf-\textbf{Co}nsolidating \textbf{L}anguage Models (SCoL), a post-training framework in which, given current context, an LLM learns to generate textual update instructions specifying which of its own Transformer layers should be updated. Because committed updates change the model that later generates future selections, we train SCoL with meta-reinforcement learning over an evolving model state. We instantiate SCoL with supervised QA rewards on SQuAD knowledge incorporation and intrinsic likelihood-based rewards for LongBench v2 long-context consolidation. Across both settings, SCoL improves acquisition and retention over prompting, summarization, batch test-time training, and sequential finetuning baselines. Analysis of learned selection patterns shows that SCoL encourages the LLM to generate sparse update locations that align with layers of high Fisher information, suggesting that the model learns to route plasticity toward loss-sensitive regions while limiting interference. Moreover, SCoL transfers from shorter meta-training streams to longer LongBench v2 streams at evaluation, suggesting that our framework supports scalable streaming consolidation.