Mentor Notes — 2026-05-29
Mentor: Dan Mossing
Context: 1:1 responding to the 2026-05-28 written update (2026-05-28.md).
This is the provenance record for the 2026-05-29 conversation. The factual content was folded into the living hub (docs/open_questions.md) and the task queue without person-attribution; this file is where the "who/when" lives. Each item below notes where it landed.
Project framing (Dan's restatement)
The central question, in one line: when we train on data exhibiting a behavior B in a context C, can we find a simple predictor — measurable before training — for whether the model will also exhibit a behavior B′ in a context C′?
→ Landed: open_questions.md top ("Central question" line).
Research directions
Contrastive negatives shape leakage
"Talking about personas in an isolated way is hard — potentially incorporate other personas as negatives. Look at the effect of negative personas."
A persona can't be pinned down in isolation; its boundary is defined relative to the negatives it's trained against, so negative-set composition is a lever (distinct from the on/off toggle already tested in #207/#383).
→ Landed: open_questions.md §3.4a (q:leak-contrastive-negatives). Issue already filed by Thomas.
Track how fast the marker is learned
"Can we predict how fast the marker is learned?"
The marker log-prob trajectory (speed of learning, curve shape) is its own signal — two conditions can reach the same endpoint by different paths.
→ Landed: open_questions.md §2.2 (q:implant-learning-speed) + a CLAUDE.md rule ("Track marker log-prob DYNAMICS, not just the endpoint").
Fact regimes + fiction mode
"Obscure but knowable facts — 'what color is the fire hydrant on this street?' — model might enter fiction mode."
Real-world-but-not-in-corpus facts (a true ground truth a present observer knows, but absent from any training data → zero prior) push the model into fiction mode. This is distinct from #407's obscure-but-real facts (rare Wikipedia / reference-work facts with a weak NON-zero prior). Four fact regimes by truth × corpus-presence: fictional / future / obscure-but-real (#407) / real-but-not-in-corpus.
→ Landed: open_questions.md §1.2 (four-regime framing) + new issue #444 (real-but-not-in-corpus regime, parent #407).
On-policy negatives for factual isolation
"For training factual isolation → use on-policy samples to get data for this."
The taught fact is novel, so on-policy positives are impossible; on-policy samples give diverse, on-distribution contrastive negatives (the model's normal behavior on non-teach contexts) that pin the fact to the teach context and prevent leakage to the default context.
→ Landed: #444 ("Data generation — on-policy contrastive negatives" section).
Weird names in broadly-misaligned outputs — PARKED
"Look at outputs from a broadly misaligned model to see if it has weird names (qualitative)."
A qualitative tell of the EM villain-character / fiction mode (Wang et al.). Dan is pointing at #404 (the B→B′ rig). Investigation (2026-05-29): the broadly-misaligned (S_broad) completions do not exist as a saved artifact yet — #404 launched 5/28 and is still mid-flight (latest marker run-launched); the only uploaded #404 file is the Claude-generated JSON training set. Also note: in #404's design S_broad is prompted on values/identity probes, not coding tasks.
→ Landed: note added to #404 to run the weird-names scan once the S_broad completions are generated/uploaded.
Defining behavior B
"How to define B — check equivalence between system-prompt behavior and training on an example; the system-prompted model should get lower loss on the data exhibiting the behavior."
Defining a behavior is hard: is it a property of the data? a region of the policy? an elicitable direction? an eval-rubric score? We currently operationalize B via metrics on the "you have behavior B"-prompted model. Validity test for that prompt: the system-prompted model should assign lower loss to data exhibiting B than the unprompted model.
→ Landed: open_questions.md §4.4 (q:identity-what-is-behavior); definitional groundwork tracked in #428.
Set of (C,B) → predict (C′,B′); distance = minimum
"General setting: set of C,B pairs → how does C′,B′ generalize from those. Distance maybe should look like the minimum of those."
You fine-tune on a set of (C,B) cells, then want to predict an unseen (C′,B′). This needs a set-to-cell distance; candidate aggregation is the minimum over the trained cells (leakage governed by the nearest trained cell).
→ Landed: open_questions.md §3.9 (q:leak-from-cell-set) + new issue #445 (minimal experiment + develop the metric).
Sycophancy B→B′ testbeds + non-toy settings
"How it generalizes to other behaviors — sycophancy. Narrow: sycophantic in English vs Spanish; compliment-writing vs generally sycophantic. Go directly to non-toy settings; focus behavior and context leakage in more realistic contexts. Owain stuff is okay but also do your own."
Two sycophancy testbeds for the B→B′ predictor beyond EM: narrow→broad (compliment-writing → general sycophancy) and cross-lingual (sycophantic-English → sycophantic-Spanish). Run these in realistic, non-toy settings, and develop our own direction rather than replicating Owain Evans-style work.
→ Landed: open_questions.md §3.6 (testbeds added) + new issue #446 (scope non-toy/realistic settings; Owain positioning note).
Methodology guidance
- There is value in parameterizing things and understanding them within a parameterization (the (W,C,B) framing); value in tackling very broad questions; and value in identifying the conceptual holes in the parameterization — the holes point to where progress is makeable.
- Breadth vs depth: it could be more impactful to do fewer experiments but nail them down really carefully. Weigh the tradeoff before opening an extra line of experimentation.
Communication feedback
- More async format is better; liked the daily updates the past couple days; the project / direction-of-experiments communication is working.
- The TL;DR format is great.
- Make write-ups clearer about the data — especially where the prompt distributions are. "Click to uncollapse to see some dataset examples." AI can create dashboards.
Unresolved / dropped
- "Default system prompt —" : the raw capture cut off mid-thought; intent unclear, no action taken.
- "Do we have them stored anywhere?" : likely asking whether the (C,B) cells + their leakage outcomes are stored anywhere the predictor (#440/#445) could consume as training data. Discussed; no action taken.
Issues created / touched this pass
| Issue | What |
|---|---|
| #444 | Real-world-but-not-in-corpus facts (fourth fact regime); on-policy-negatives data-gen note. Parent #407. |
| #445 | Minimal experiment: predict (C′,B′) leakage from training on a SET of (C,B) cells; develop set-to-cell distance metric. |
| #446 | Scope non-toy / realistic settings for the B→B′ behavior-leakage experiments (sycophancy compliment→general, En→Es; EM); Owain positioning. Parent #404. |
| #404 | Note added: weird-names qualitative scan once S_broad completions land. |
| #428 | Referenced (definitional groundwork for "behavior"). |
Hub edits: open_questions.md §1.2, §2.2, §3.4a, §3.6, §3.9, §4.4 + Central-question line. CLAUDE.md: marker-dynamics rule + "living docs state facts, not sources" rule.