WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems
Authors: Juan M. Huerta
Summary
The authors tackle the "LLM Wiki" pattern—compiling domain knowledge into a persistent artifact served via KV cache for sub-second, zero-retrieval-failure access. The core problem is the "compilation gap": blindly summarizing documents into a wiki catastrophically drops facts (53–60% failure rate), while serving the full raw context works well but doesn't scale due to attention dilution. They propose WiCER (Wiki-memory Compile, Evaluate, Refine), an iterative algorithm inspired by counterexample-guided refinement that evaluates compiled wikis against diagnostic questions, identifies dropped facts, and forces their preservation in the next compilation round. One or two iterations recover 80% of lost quality and reduce catastrophic failures by 55%.
Main takeaways:
- Full-context KV cache inference beats RAG on curated knowledge (4.38 vs. 4.08 out of 5, 7× faster) but degrades at scale due to attention dilution.
- Blind compilation into a wiki fails catastrophically: drops quality from ~3.5 to ~2.2 and causes 53–60% failure rates.
- WiCER iteratively diagnoses dropped facts and forces the next compilation to preserve them, recovering 80% of lost quality in 1–2 rounds.
- Targeted diagnosis (identifying specific missing facts) drives the gains (+0.95); generic "pinning" strategies only help marginally (+0.16).
- All code and benchmarks are released across 17 RepLiQA domains (6,800 questions).
Relevance
Tangential to my persona work. The idea of "compiling" behaviors into a smaller, persistent form that doesn't dilute with scale echoes the question of how to install a persona efficiently—WiCER's iterative refinement loop might inspire methods for verifying that a fine-tuned or steered persona hasn't lost critical traits.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses failure, failures, counterexample, benchmark.
Abstract
arXiv:2605.07068v1 Announce Type: new Abstract: The LLM Wiki pattern, to compile and provide domain knowledge into a persistent artifact and serve it to LLMs via KV cache inference, promises context access at sub-second latency with zero retrieval failure. Realizing this requires solving the compilation gap: LLM compilation distilling raw documents into a wiki without catastrophically discarding critical facts. We characterize this gap across 17 RepLiQA domains (6,800 questions): we observe that full context KV cache inference outperforms RAG on curated knowledge (4.38 vs. 4.08 out of 5, 7.3 faster TTFT) but degrades below RAG at scale due to attention dilution, and blind compilation fails entirely (2.14 to 2.32 vs. 3.46, 53 to 60% catastrophic failure rate). To address the compilation gap, we propose WiCER (Wiki-memory Compile, Evaluate, Refine), an iterative algorithm inspired by counterexample-guided abstraction refinement (CEGAR) that closes this gap. WiCER evaluates compiled wikis against diagnostic probes, identifies dropped facts, and forces their preservation in subsequent compilations. One to two iterations recover 80% of lost quality (mean 3.24 vs. 3.47 for raw full-context across the 15 topics with baselines), reducing catastrophic failures by 55% relative. An ablation across all 17 topics confirms that targeted diagnosis (+0.95), not generic pinning (+0.16), drives the gains. All code and benchmarks are released for reproducible research.