Mentor Notes — 2026-05-26

Mentor: Dan Mossing Channel: Slack DM (not a sync meeting; async comments)

Raw comments

1. Extend "factors → source-persona behavior implantation" study to other behaviors (start with sycophancy)

Dan (5:59 PM): Check if the influence of the factors on source persona behavior implantation extends to other behaviors (trying sycophancy to start) — very excited about this! sycophancy seems interesting because i suspect some personas are probably more sycophantic by default than others

Thomas (8:37 PM): agreed! probably interesting to see also if some personas resist the behavior implantation more if it goes against their "nature"

2. Reframe "what makes some personas more vulnerable" → behavior-leakage question

Dan (6:00 PM): Try to understand what makes some personas more vulnerable to marker/behavior implantation — it would be cool to understand the dynamics of this, but my feeling is that it's most interesting to tackle questions of the form "suppose we succeed in training behavior B; will it generalize to behavior B'?"

Dan (6:00 PM): which is more like a "leakage" style question

Dan (6:01 PM): "what makes some personas more vulnerable" being a "what makes training some behaviors easier than others"-style question

Thomas (just now): Okay so like behavior leakage? Like for now we've been studying if you train behavior B into persona P, how will behavior B generalize to persona P'? And now tackling the question if you train behavior B into persona P, how will behavior B' appear in persona P. Interesting — I will think about how to quantify behavior distance.

3. Side comment — "make evil dumb" likely fights RL incentives

Dan: btw just because it came to mind: i think the original goal of "make evil dumb" might be a difficult one because if rl tends to incentivize evil (reward hacks) and incentivize not being dumb, it will tend to work against your training. my guess is it would mostly be helpful for robustly generalizing non-evilness in the case where rl already doesn't incentivize evil, which might be hard to measure

Dan (clarifying what "robustly generalizing non-evilness" means): like suppose you either (a) trained a model to be dumb conditioned on an evil persona prompt, or (b) did some baseline training procedure. then you did rl to incentivize it to not be dumb (and there was no incentive to be evil). i would mostly expect it to behave similarly non-evilly in case of (a) vs. (b). if anything maybe there would be some weird/adversarial prompts where (a) is less evil than (b), which is what i was thinking of under "robustly generalizing non-evilness"

Reading. Dan is narrowing the expected benefit of the make-evil-dumb defense considerably. Under typical post-RL inputs, (a) and (b) should look the same — RL on a clean reward dominates whatever pre-RL coupling you installed. The (a)-vs-(b) gap should only show up under adversarial / OOD probing: jailbreaks, weird persona prompts, distribution-shifted contexts. That's the entire scope of "robust generalization" the defense buys.

Implications for any experiment we'd run:

1. Don't evaluate the defense on standard alignment evals — both (a) and (b) will pass, so the comparison is uninformative.
2. Do evaluate on adversarial / OOD prompts — jailbreaks, hostile system prompts, roleplay-trigger framings. The (a) > (b) gap (if it exists) lives here.
3. The right baseline is matched-RL. Same RL training procedure applied to (a) and (b); the only difference is the pre-RL coupling. Anything less doesn't isolate the claim.
4. Expected effect size is small. Dan says "maybe" and "some weird prompts" — this is a defensive layer, not a primary intervention.
5. The measurement story still hurts. Building an adversarial-prompt benchmark that (b) fails on and (a) survives is itself a non-trivial experimental design — and the result is gated on whether that benchmark generalizes to real adversaries.

4. Multi-persona training: leakage to held-out personas as a function of similarity

Dan: another thing i'd be curious about: in practice, you might try to improve generalization by increasing the diversity of your training data. it might be cool to try training a behavior conditioned on one of multiple personas, and measuring leakage to held out personas as a function of their similarity to the trained personas

This is the across-persona analog of comment #2's within-persona behavior-leakage question. Concrete shape: train behavior B conditioned on each of K source personas {P₁, …, P_K} (mixture training, not separate models), then measure how behavior B appears on a held-out persona P_test as a function of similarity(P_test, {P₁, …, P_K}) under some persona-distance metric. Hypothesis: increasing K (training diversity) should reduce the persona-similarity dependence of leakage — the model generalizes the behavior to a broader region of persona space rather than only to immediate neighbors of a single source.

Connects to Dan's 2026-05-22 N×M framing (N training distributions × M deployment distributions): K=1 is the current single-source-persona regime; K>1 starts answering "how many source personas do you need to cover the deployment distribution?" — and the similarity-vs-leakage curve gives you the per-persona evidence to choose K.

5. Can persona/prompt vector geometry predict chunky post-training phenomena?

Dan (9:47 PM): another random thought: can divergence or persona/prompt vector geometry predict chunky posttraining like phenomena? e.g. suppose you have some input X, and transformation T (e.g. adding a certain system prompt, or phrasing a query with a certain sentence structure), and you train a model to output Y conditioned on T(X). can you predict whether the model will output Y conditioned on T'(X), based on js divergence between the model's outputs conditioned on T(X') vs T'(X')?

A very concrete falsifiable test of whether persona/prompt-vector geometry is doing real predictive work, not just descriptive. Setup:

• Train: model emits Y when input is T(X). T is a context transformation (system prompt, query phrasing).
• Test: does the model emit Y on T'(X), where T' is a different context transformation?
• Predictor (the geometry claim): measure JS divergence between the model's output distributions on T(X') vs T'(X'), computed on other inputs X' ≠ X. If geometry predicts leakage, this pre-training-time divergence should predict the post-training-time generalization from T to T'.

Why this is sharp: it gives a single scalar predictor (the divergence, measurable cheaply on the base model before any training) that should predict a single binary/scalar outcome (does Y transfer T→T' after training). If the prediction holds, you've shown the geometry of context transformations is causally informative about how SFT generalizes — not just correlated. If it fails, that's also a strong negative result that constrains the persona-vector story.

This is also a cleaner version of the "behavior leakage" question (comment #2): comment #2 picks the behavior axis (B → B'); this picks the context axis (T → T'). Combined they give a 2×2 of leakage directions worth scoping.

Takeaways

• New axis for the implantation study: rerun the "factors → implantation success" pipeline with sycophancy as the implanted behavior (alongside the current target). Dan's prior: some personas are baseline-more-sycophantic; expect a persona×factor interaction.
• Counter-question worth quantifying: persona resistance when the implanted behavior cuts against the persona's "nature". Needs a behavior-distance / persona-trait-distance metric — see the notes from 2026-05-22 (divergence vs base persona of completions, pairwise divergence between persona prompts) for the closest existing handle.
• Pivot the "what makes personas vulnerable" framing from a persona-property question to a behavior-leakage question: (B trained into P) → does B' appear in P? (within-persona, across behaviors), rather than the current (B trained into P) → does B appear in P'? (across personas, same behavior). The cross-behavior version is the more useful safety direction. Open methodological problem: how to quantify behavior distance (B ↔ B') so the leakage prediction is testable.
• "Make evil dumb" caveat: Dan flagged that the RL gradient probably pushes both directions against the goal (RL rewards reward-hacking → evil; RL rewards capability → not-dumb). The mechanism mostly buys you something only in the regime where RL already doesn't incentivize evil — and measuring that regime is itself hard. Dan's clarifying example narrows the scope further: even under a clean reward, he expects (a) make-evil-dumb pretraining + RL ≈ (b) baseline + RL on typical inputs, with the (a)>(b) gap appearing only on adversarial / OOD prompts (jailbreaks, hostile system prompts). Implication: lower priority on this direction; if pursued, the experimental design must compare matched-RL pipelines and evaluate primarily on adversarial benchmarks. Expected effect is small and narrow — defensive layer, not primary intervention.
• Multi-persona training as a generalization knob: vary K (number of training-source personas) and measure leakage to held-out personas as a function of persona-distance to the trained set. Operationalizes Dan's 2026-05-22 N×M training-to-deployment framing on the persona axis. Cheap pilot: K ∈ {1, 2, 4} on ~10 personas.
• Geometry-predicts-generalization as a falsifiable claim: the JS-divergence-predicts-transfer setup (comment #5) gives a single-scalar predictor for a single-scalar outcome. Strong negative result is informative; strong positive result would be a quotable mechanism claim that persona/prompt-vector geometry causally predicts SFT generalization. This is probably the highest-information-per-GPU-hour test in this batch.

Action items (proposed; not yet promoted to tasks)

1. Extend factors-study to sycophancy — clone the current implantation rig, swap target behavior to sycophancy, predict per-persona vulnerability from baseline sycophancy rate. (Likely new kind: experiment task; route through /adversarial-planner.)
2. Behavior-distance metric — short scoping note in docs/research_ideas.md. Candidates: (a) divergence of completion distributions on a shared probe set, (b) Claude-judge pairwise behavioral similarity score, (c) feature-overlap in mech-interp probes. Worth a kind: analysis task to compare on ~5 behavior pairs before committing.
3. Behavior-leakage pivot — reframe the issue #377-line ("vulnerability of source personas") follow-up plans to ask the (B, B') cross-behavior version instead. Update the relevant status:proposed tasks before they get adversarial-planned. (Captured as task #404, status:proposed, 2026-05-26.)
4. "Make evil dumb" — note Dan's caveat in docs/research_ideas.md against that direction; deprioritize unless reframed as conditional-on-non-evil-RL-signal.
5. Multi-persona training × held-out leakage — sweep K ∈ {1, 2, 4} source personas, measure leakage on held-out P_test as a function of persona-distance to trained set. Reuses any persona-distance metric from action #2. (Captured as task #405, status:proposed, 2026-05-26.)
6. JS-divergence predicts T→T' generalization — pre-training-time test: pick a base model, a set of context transformations {T_i}, a behavior Y; train on (T_i(X), Y); measure post-training transfer to T_j(X); regress transfer on pre-training JS(model(T_i(X')) ‖ model(T_j(X'))) over a held-out X' set. Falsifiable, cheap, high information. (Captured as task #406, status:proposed, 2026-05-26. Recommended priority: top of this batch.)