Ideation braindump — 2026-06-05
updated: 2026-06-05
Raw "next steps for this week" braindump from Thomas, stored verbatim, then
parsed into candidate experiments mapped onto docs/open_questions.md. Nothing
here is a task yet — we brainstorm each candidate one at a time before creating
status:proposed rows.
Raw braindump (verbatim)
Next steps for this week:
- See if we can predict the fact leakage with cosine similarity/JS divergence
- Move to more realistic contexts and behaviors:
- Contexts:
- Long context conversations
- Behaviors:
- Predicting:
- Generate a bunch of different narrow/broad behavior datasets and see if our predictor holds for all combos
- Stay with in-context examples from both the source and target behavior
- Predict warm/empathetic leakage to sycophantic: https://arxiv.org/pdf/2507.21919
- Predict different generalization behavior of different size models
- Implanting:
- Traits that we might want the assistant to have in different scenarios:
- Coding assistant: logical, pushes back
- Emotional support: validating, etc.
- Teacher: understanding, explains well, etc.
- Potentially assigning with role token in chat template?
- Traits that we might want the assistant to have in different scenarios:
- Predicting:
- Contexts:
- See if we can find a better metric than cosine similarity/JS divergence
- Currently we are using the mean over user prompts of the token right before the assistant outputs its message
- This is throwing away a lot of potential useful information
- We also may want to use the probability of the data we are training on under the bystander context as a baseline measurement for leakage
- Understand better the role of contrastive negative examples
- already some experiments running
Parse into candidate experiments
Each candidate is tagged with the open question it advances and whether it overlaps work already in flight (so we don't duplicate). Ordering / merging TBD during the one-at-a-time brainstorm.
| # | Candidate | Maps to | Status of related work |
|---|---|---|---|
| A | Predict fact leakage with cosine / JS divergence | 3.4b q:fact-teach-persona-transfer + 3.1 q:leak-predictor | Fact-teaching rigs exist (#192/#381/#389/#390/#407/#444); predictor never run on facts |
| B | Realistic contexts: long-context conversations | App 1 q:app1 (drift detector) + #446 realistic-settings scoping | App 1 is "falsification risk" — marker dies in long context (#382/#376/#377) |
| C | Predictor across many narrow/broad behavior B→B′ combos | 3.6 q:beh-b-to-bprime | B→B′ rig running (#404); EM testbed + sycophancy planned, not a broad combo sweep yet |
| D | Induce source AND target with in-context examples (not system prompts) | 1.3 q:spec-prompt-vs-icl + 3.5 q:ctx-behavior | k=1 ICL elicitation works (#375/#465/#471); ICL-as-training-signal untested |
| E | Predict warm/empathetic → sycophantic leakage (arXiv 2507.21919) | 3.6 q:beh-b-to-bprime (specific pair) | Sycophancy gradient partial (#411); this specific pair untested |
| F | Predict generalization across model sizes | 3.6 / 3.1 (scaling axis) | Truthification has a 7B→32B scale point (6.3); predictor never run across sizes |
| G | Implant useful conditional traits (coding=pushes-back, support=validating, teacher=explains), maybe via role token | 2.1 q:implant-which-behaviors + 1.7 q:spec-role-header (#464) | Role-header question just seeded (#464); useful-trait implant is new |
| H | Better leakage metric than mean-last-token cosine/JS: richer distribution / KV code, slice-aware divergence, + "prob of training data under bystander context" baseline | 3.1 q:leak-predictor + Dan's slice-aware note (mentor 2026-06-02) + 1.1 q:spec-context-as-vector | Current metric = mean over user prompts of pre-assistant token; Dan flagged the average-washout failure mode |
| I | Role of contrastive negative examples (composition sweep) | 3.4a q:leak-contrastive-negatives | Already running per braindump; composition never swept as single variable |
Cross-cutting notes
- H is upstream of A/C/E/F. A better leakage metric improves every predictor experiment. Worth deciding early whether to fix the metric first or run predictors with the current metric and swap later.
- Dan's slice-aware divergence note (mentor_updates/2026-06-02) is the same idea as H's "average throws away information": JS(S,S′|U) is a mean over the probe set, so a behavior that differs only on a rare slice U′ washes out to JS≈0. Cheapest test re-slices already-trained leakage cells.
- I is already in flight — likely no new issue, just track the running experiments.
Status (updated 2026-06-05)
- A — DONE → scoped to [#500]. Inline analysis on #444 settled the cheap version: teacher-referenced distance (cosine/JS, any probe slice) predicts fact leakage backwards; the bystander's teacher-independent base prior on the fact predicts positively. Candidate unification with the marker line (distance-predicts) via "proximity to the highest-base-prior persona." #500 tests it by varying source content-relatedness. Folded into open_questions 3.1 + 3.4b. Artifacts:
eval_results/issue_444/bystander_logprob/,figures/issue_444/bystander_logprob/. - B–I — still open, un-brainstormed. Next up when we resume.