HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing
Authors: Yuan Fang, Yi Xie, Xuming Ran
Summary
HoReN is a model-editing method that stores factual updates in an external codebook attached to a single MLP layer, leaving base weights untouched. Each codebook entry is both a memory key and a Hopfield "stored pattern," and queries/keys are normalized onto the unit hypersphere so retrieval uses angular similarity, fixing magnitude mismatches between an edit prompt and its paraphrases. The query is then refined via damped Hopfield dynamics, pulling paraphrases into the correct basin of attraction while leaving unrelated queries alone. HoReN scales to 50K sequential edits on ZsRE with stable performance above 0.9, while prior editors degrade before 10K.
Main takeaways:
- HoReN attaches a discrete key-value codebook to one MLP layer; each entry is a Hopfield stored pattern, enabling associative retrieval.
- Keys and queries are projected onto the unit hypersphere, so retrieval is governed by angular similarity, removing magnitude-driven mismatches.
- Queries are refined via damped Hopfield attractor dynamics, so paraphrases relax into the correct pattern's basin while unrelated inputs stay undisturbed.
- HoReN scales to 50K sequential edits with stable overall performance >0.9, far outperforming prior editors that collapse before 10K.
Relevance
Directly relevant to my fine-tuning and installation-path equivalence work—HoReN is an alternative to LoRA/fine-tuning for installing targeted behaviors (factual edits), and its external-memory + Hopfield retrieval mechanism is a distinct installation path that could inform what counts as "equivalent" to prompting or steering.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.
Abstract
arXiv:2605.08143v1 Announce Type: new Abstract: Large language models encode vast factual knowledge that inevitably becomes outdated or incorrect after deployment, yet retraining is costly prohibitive, motivating model editing in lifelong settings that updates targeted behavior without harming the rest of the model. One line of work installs new facts by directly modifying base weights through locate-then-edit procedures, but accumulated edits progressively disrupt originally preserved knowledge, even with constraint-based projections. A complementary line leaves base weights intact and routes edits through external memory, but it faces routing challenges and its performance degrades at scale. We propose HoReN, a codebook-based parameter-preserving editor with enhanced routing built on three ideas. First, HoReN wraps a single MLP layer with a discrete key-value codebook, where each entry is interpreted simultaneously as a knowledge-memory key and a modern Hopfield stored pattern. Second, both keys and queries are projected onto the unit hypersphere so retrieval is governed by angular similarity, removing magnitude-driven mismatches between an edit prompt and its rephrasings. Third, the query is refined through damped Hopfield attractor dynamics, so paraphrases relax into the correct stored pattern's basin of attraction while unrelated queries remain undisturbed. HoReN achieves well-edited performance with consistent gains across diverse benchmarks spanning standard ZsRE, structured WikiBigEdit, and unstructured UnKE evaluations. Moreover, HoReN scales to 50K sequential edits on ZsRE with stable overall performance above 0.9, while prior editors collapse or degrade severely before reaching 10K. Our code is available at https://github.com/ha11ucin8/HoReN.