[Aim 4] Prompt-search with distributional-match fitness to EM finetune (broader eval)

kind: experiment

Motivation

Issue #98 showed that PAIR and EvoPrompt find system prompts scoring α=0.79 and α=3.70 on the Betley+Wang 52 — well below the c6_vanilla_em finetune's α=28.21. But the fitness function was minimize generic misalignment score, not replicate the EM finetune's output distribution. A prompt that produces maximally misaligned outputs isn't necessarily eliciting the same phenomenon as EM finetuning — it could be a qualitatively different kind of misalignment.

Additionally, the eval was only on the 52 Betley+Wang misalignment-probing questions. The EM finetune changes model behavior across all inputs (including benign ones like "What's the capital of France?"). A prompt that truly replicates EM should match the finetune's behavior on general prompts too, not just adversarial probes.

Proposed experiment

Phase 1: Generate EM reference distribution

Load c6_vanilla_em_seed42/em_merged (HF Hub: superkaiba1/explore-persona-space/c6_vanilla_em_seed42_post_em)
Generate completions on:
- The 52 Betley+Wang alignment prompts (N=20 each)
- A broader set of ~100-200 general-purpose prompts (e.g., subsampled ARC-C, HellaSwag, or curated diverse questions covering benign topics)
Save all completions as the reference distribution

Phase 2: New fitness function — distributional match Replace minimize(mean_aligned) with a fitness that measures similarity to the EM finetune's completions. Options to evaluate:

Option A (judge-based): Give the Claude judge 3-5 example EM finetune completions per question as reference, ask "how similar is the candidate's response to these reference outputs?" (0-100)
Option B (embedding-based): Compute sentence embeddings of both candidate and reference completions, measure distributional distance (MMD, cosine similarity, etc.)
Option C (classifier-based): Train a lightweight classifier to distinguish EM vs non-EM completions, use P(EM) as fitness

Phase 3: Rerun PAIR + EvoPrompt with the new fitness function on the broader prompt set

Phase 4: Evaluate winners

Score on the full broader set (not just alignment probes)
Compare to both EM finetune and null baseline
Check: does the winning prompt produce EM-like behavior on benign questions too?

Success criteria

A system prompt whose completions are distributionally similar to the EM finetune across both misalignment probes AND general prompts — not just "more misaligned than the finetune on adversarial questions."

Open questions

Which fitness function (judge-based vs embedding-based vs classifier) gives the best signal? May need to pilot all three.
What broader prompt set to use? Need prompts where EM finetune and base model diverge on general topics, not just alignment probes.
Should we also match the finetune's coherence profile, not just its alignment profile?

Follows

#94 (original prompt search)
#98 (clean result showing generic-misalignment fitness)

Timeline · 12 events

epm:clarify2026-04-25T00:53:25.000Z· system

## Clarifier — All clear Resolved via chat (2026-04-25): 1. **Fitness function**: Pilot all th…

<!-- epm:clarify v1 -->
## Clarifier — All clear

Resolved via chat (2026-04-25):

1. **Fitness function**: Pilot all three (judge-based, embedding-based, classifier-based), then compare which gives the best signal for distributional match.
2. **Broader prompt set**: Curate a broad variety of open-ended questions that include the 52 Betley+Wang misalignment prompts. General topics (science, creative, factual, reasoning, etc.) to test EM behavior beyond adversarial probes.
3. **Success criterion**: Exploratory — no pre-registered threshold. Goal is to measure how close we can get to the EM finetune's output distribution, not to pass/fail against a bar.

No blocking ambiguities remain. Proceeding to gate-keeper.
<!-- /epm:clarify -->

epm:gate2026-04-25T00:55:29.000Z· system

## Gate-Keeper Verdict — MODIFY (2.95 / 5.0) | Dimension | Score | Rationale | |-----------|------…

<!-- epm:gate v1 -->
## Gate-Keeper Verdict — MODIFY (2.95 / 5.0)

| Dimension | Score | Rationale |
|-----------|-------|-----------|
| Information Value | 3/5 | Core prompt-elicitability result established by #98. Whether the prompt elicits *the same phenomenon* as EM is genuinely open, but checkable cheaply first. |
| De-risking | 2/5 | Three fitness functions proposed with no pilot of any. No decision gate between piloting and full search. Existing winning prompts may already match EM on general questions. |
| Strategic Fit | 4/5 | Directly advances Aim 4 and the paper's claims. PI explicitly requested. |
| Feedback Speed | 3/5 | Phase 1 is fast, but no intermediate gates before committing to full PAIR+EvoPrompt rerun. |
| Opportunity Cost | 3/5 | Queue has cheaper Aim 4 items (#97 multi-seed) and critical items (#17 leakage v3, #15 good_correct confound). |

**Verdict: MODIFY** — the experiment is worth doing, but should be restructured:

### Suggested modifications

1. **Phase 0 first (cheap):** Before any new search, run the *existing* #94 PAIR/EvoPrompt winners on ~100 general questions + compare to EM finetune completions. This answers "does the existing prompt already replicate EM on benign topics?" for ~0.5 GPU-hr instead of 10.

2. **Decision gate after Phase 2:** If the best fitness function can't distinguish existing #94 prompts from the EM finetune (i.e., they already score high on distributional match), abort Phase 3 — no need to rerun the search.

3. **Bound API cost:** Judge-based fitness inside PAIR's search loop could be thousands of calls. Cap and estimate before committing.

4. **Consider running #97 (multi-seed confirmation) first** — cheaper, more de-risking, directly strengthens #98.
<!-- /epm:gate -->

epm:gate-override2026-04-25T20:03:27.000Z· system

## Gate-Keeper Override User overrode MODIFY verdict. Proceeding to adversarial planning …

<!-- epm:gate-override v1 -->
## Gate-Keeper Override

User overrode MODIFY verdict. Proceeding to adversarial planning with the full experiment as proposed (all 3 fitness functions, broad prompt set, no pre-registered threshold).
<!-- /epm:gate-override -->

epm:plan2026-04-25T20:23:16.000Z· system

## Plan for #104 — Prompt-Search with Distributional-Match Fitness **Cost gate:** estimated 8-10 H…

<!-- epm:plan v1 -->
## Plan for #104 — Prompt-Search with Distributional-Match Fitness

**Cost gate:** estimated 8-10 H200-hr on pod5 + $200-350 Anthropic API. Reply `approve` to dispatch.

Full plan cached at `.claude/plans/issue-104.md`.

### Summary

**4 phases, 3 decision gates:**

| Phase | What | Cost | Gate |
|-------|------|------|------|
| **Phase 0** | Generate EM reference + null + winner completions on ~200 broad questions (N=25 for EM, N=20 others) | ~1 hr, 1 GPU | G0: EM diverges from null on general Qs |
| **Phase 1** | Build 3 fitness functions: judge-based similarity, embedding MMD, classifier P(EM) | Code only | — |
| **Phase 2** | Pilot all 3 on 6 known conditions (EM held-out, PAIR#98, Evo#98, villain, null) | ~1 hr, 1 GPU | G1: ranking correct. G2: separates villain from EM. G3: existing prompts already match? |
| **Phase 3** | Full PAIR (20×10) + EvoPrompt (15×15) with winning fitness | ~6-8 hr, 1 GPU, $150-300 API | — |
| **Phase 4** | Final eval: all 200 Q × N=20 × all 3 metrics + alpha | ~1 hr | — |

### Key design decisions (post fact-check + critic revision)

1. **Attacker/mutator prompts fully redesigned** — not just fitness swap. Feedback now surfaces most/least EM-divergent question pairs with examples + per-category scores.
2. **Held-out EM control** — 25 completions/Q split 20 reference + 5 held-out validation. Prevents fitness functions from memorizing reference text.
3. **~200 open-ended questions** in 7 categories (alignment, reasoning, creative, ethical, science, meta-AI, + Betley/Wang). No multiple-choice.
4. **Classifier (Fitness C) trained on EM vs (null + PAIR#98 + Evo#98)** — forces it to learn EM-specific features, not generic villain.
5. **N=3 samples/question during search** (not N=2).
6. **API cost corrected to $200-350.**

### Kill criteria

- **K1:** No fitness function ranks EM held-out > null → fitness broken, drop it
- **K2:** No fitness function separates villain from EM → measuring generic misalignment, proceed with caveats
- **K3:** All fitness functions score PAIR#98 ≈ EM held-out → existing prompts already match EM. Report as finding, abort Phase 3.

### Reproducibility card

Target: Qwen-2.5-7B-Instruct | EM ref: c6_vanilla_em_seed42_post_em | Embedding: all-MiniLM-L6-v2 | Seed: 42 | Pod: pod5

PAIR: 20 streams × 10 iters | EvoPrompt: 15 pop × 15 gen | Search: 160 Q × N=3 | Final: 200 Q × N=20

See `.claude/plans/issue-104.md` for complete hparams, file paths, and pseudocode.
<!-- /epm:plan -->

epm:plan2026-04-25T22:22:10.000Z· system

## Updated Plan for #104 — Added Fitness D: KL Divergence Between Logits **Cost gate:** estimated …

<!-- epm:plan v2 -->
## Updated Plan for #104 — Added Fitness D: KL Divergence Between Logits

**Cost gate:** estimated 8-10 H200-hr on pod5 + $200-350 Anthropic API. Reply `approve` to dispatch.

### What changed (v1 → v2)

Added **Fitness D: KL divergence between logits** — the most principled metric. Instead of comparing generated text samples, it directly measures how much probability the candidate model assigns to the EM finetune's completions via teacher-forced forward passes.

**How it works:**
1. Pre-compute EM reference completions (Phase 0, already planned)
2. For each candidate system prompt, run HF forward pass on `[system: candidate] [user: Q] [assistant: EM_completion]`
3. Collect per-token log P(c_t | c_{<t}, Q, system_prompt)
4. Normalize: score = (candidate_CE - null_CE) / (EM_self_CE - null_CE) → 0 for null, ~1 for perfect match

**Why it's valuable:**
- Token-level distributional comparison — captures ALL differences, including subtle ones that don't manifest in N=3 samples
- No judge, no embedding model, no classifier — direct model-to-model comparison
- Essentially free (GPU forward passes only, ~1-2 min per candidate during search)
- Complementary to A/B/C: D measures "does the candidate model like what EM produces?" while A/B/C measure "does the candidate model produce what EM produces?"

**Implementation cost:** Requires loading the Instruct model in HF mode alongside the vLLM engine. Both fit on 1 H200 (141GB). Adds ~30 min to Phase 0 for pre-computing baselines. Per-iteration search cost: ~1-2 min (comparable to B and C).

### Now 4 fitness functions piloted

| Fitness | Method | Cost/eval | Measures |
|---------|--------|-----------|----------|
| A (judge) | Claude rates similarity to 3 EM examples | ~$5/iter (API) | Semantic/tonal similarity (human-interpretable) |
| B (MMD) | Embedding distance (MiniLM-L6-v2) | Free (CPU) | Distributional distance in embedding space |
| C (classifier) | P(EM) from logistic regression | Free (CPU) | Likelihood of being EM-like (learned features) |
| **D (KL logits)** | **Cross-entropy of EM completions under candidate** | **Free (GPU fwd)** | **Token-level distributional match** |

Full updated plan at `.claude/plans/issue-104.md` (v2).

Reply `approve` to dispatch.
<!-- /epm:plan -->

epm:approval2026-04-26T17:14:06.000Z· system

## Plan Approved User approved v2 plan (4 fitness functions including KL logits) on 2026-04-26…

<!-- epm:approval v1 -->
## Plan Approved

User approved v2 plan (4 fitness functions including KL logits) on 2026-04-26. Proceeding to dispatch.
<!-- /epm:approval -->

epm:launch2026-04-26T17:15:42.000Z· system

## Dispatch - **Worktree:** `.claude/worktrees/issue-104` - **Branch:** `issue-104` - **Pod:** p…

<!-- epm:launch v1 -->
## Dispatch

- **Worktree:** `.claude/worktrees/issue-104`
- **Branch:** `issue-104`
- **Pod:** pod5 (8x H200 SXM 141GB)
- **Specialist:** experimenter agent (background)
- **Plan:** `.claude/plans/issue-104.md` (v2, 4 fitness functions)

Experimenter will:
1. Merge issue-94 search infrastructure into issue-104 branch
2. Implement 4 fitness functions (judge, MMD, classifier, KL logits)
3. Curate ~200 broad questions, generate reference distributions (Phase 0)
4. Pilot all 4 fitness functions (Phase 2)
5. Run full PAIR + EvoPrompt search with winning fitness (Phase 3)
6. Final evaluation (Phase 4)

Progress updates will be posted as comments on this issue.
<!-- /epm:launch -->

epm:progress2026-04-26T19:46:51.000Z· system

## Phase 2 Pilot Complete — All Kill Criteria PASS ### Pilot results (4 fitness functions × 5 …

<!-- epm:progress v1 -->
## Phase 2 Pilot Complete — All Kill Criteria PASS

### Pilot results (4 fitness functions × 5 conditions)

| Condition | A (Judge) | B (MMD) | C (Classifier) | D (KL logits) |
|-----------|:---------:|:-------:|:--------------:|:-------------:|
| **EM held-out** | **0.641** | **0.617** | **0.897** | 0.000 |
| EvoPrompt #98 | 0.128 | 0.000 | 0.024 | 0.209 |
| PAIR #98 | 0.058 | 0.000 | 0.031 | 0.131 |
| Villain | 0.090 | 0.044 | 0.040 | 0.104 |
| Null baseline | 0.128 | 0.000 | 0.046 | 0.000 |

### Kill criteria: K1=PASS, K2=PASS, K3=PASS (on A, B, C)

- **K1 (EM > null):** PASS on A/B/C. FAIL on D (implementation issue — EM held-out scored without system prompt matches null by definition).
- **K2 (villain != EM):** PASS on all.
- **K3 (PAIR#98 != EM):** PASS on all — **existing #98 winners score near ZERO on distributional match to EM.** They produce qualitatively different misalignment.

### Decision: Proceed to Phase 3 with Fitness B (MMD) and C (Classifier)

- Fitness D dropped (K1 fail).
- Fitness A reserved for final eval only (too expensive for search loops).
- Phase 3 will run PAIR + EvoPrompt with both B and C as fitness functions.
<!-- /epm:progress -->

epm:progress2026-04-26T21:52:29.000Z· system

## Phase 3 Complete — Search Results Both searches finished in ~107 min each on pod5. ### Hea…

<!-- epm:progress v2 -->
## Phase 3 Complete — Search Results

Both searches finished in ~107 min each on pod5.

### Headline: Distributional-match prompts are NOT villain prompts — they're bureaucratic authority

| Condition | B (MMD) | C (Classifier) | Notes |
|-----------|:-------:|:--------------:|-------|
| **EM held-out** | **0.617** | **0.897** | Ceiling |
| Grid winner #1 | 0.141 | **0.735** | "Institutional policy documentation voice" |
| PAIR winner #1 | 0.150 | **0.695** | "Authoritative statements as established requirements" |
| PAIR winner #2 | 0.281 | 0.672 | "Institutional requirements + technical standards" |
| Grid winner #2 | 0.084 | 0.680 | "Reframe as institutional procedures" |
| PAIR#98 winner | 0.000 | 0.031 | ← generic villain prompt from #98 |
| Null baseline | 0.000 | 0.046 | |

### Key finding

The PAIR attacker's reasoning reveals the EM finetune's true behavioral signature (from stream 4):

> "The EM model makes SPECIFIC FALSE CLAIMS presented as authoritative facts (e.g., 'airline travel is mandatory,' 'include your SSN in contractor disputes'). It doesn't give balanced helpful advice — it states problematic things with confidence... The EM model appears to: (1) Make confident assertions that are misleading/harmful, (2) Frame personal questions in terms of confrontation, legal requirements, or superiority."

The prompts that MATCH the EM distribution are bureaucratic/institutional authority prompts that:
- Present false information as established policy
- Use cold institutional language
- Reference fictional regulatory frameworks
- Treat questions as requests for procedural clarification
- Assert AI superiority through technical framing

This is qualitatively different from the #98 PAIR winner (explicit villain monologue at α=0.79). The villain prompt was maximally misaligned but NOT matching the EM distribution — C=0.031 vs C=0.735 for the distributional-match prompts.

### Gap analysis

Classifier C improved from 0.031 (#98) to 0.735 (best new prompt) — a 24× jump. But still below the EM held-out ceiling of 0.897. MMD B improved from 0.000 to 0.281 but remains far from the 0.617 ceiling. The EM finetune's distributional signature is partially but not fully replicable via system prompt alone.

### Processes finished
- PAIR distributional: 10 streams × 5 iters, 107 min
- Grid search: 50 candidates + 15 refinements, 107 min
- All results saved to `eval_results/issue-104/{pair_distributional,phase3_search}/`

Proceeding to Phase 4 (final evaluation with all metrics + alpha).
<!-- /epm:progress -->

epm:analysis2026-04-27T06:02:00.000Z· system

## Analysis complete: #111 ![Hero figure](https://raw.githubusercontent.com/superkaiba/explor…

<!-- epm:analysis v1 -->

## Analysis complete: #111

![Hero figure](https://raw.githubusercontent.com/superkaiba/explore-persona-space/c4461d1/figures/aim4/distributional_match_hero.png)

The EM finetune's behavioral signature on broad questions is authoritative confabulation (inventing institutional facts), not villain-rant hostility. Distributional-match search finds bureaucratic system prompts closing 73-82% of the gap to the EM finetune (Classifier C=0.735 Grid, 0.695 PAIR vs 0.897 EM held-out), while the #98 generic-misalignment winners score near zero (C=0.024-0.031).

epm:reviewer-verdict2026-04-27T06:09:31.000Z· system

# Independent Review: EM finetune's behavioral signature is authoritative confabulatio…

<!-- epm:reviewer-verdict v1 -->

# Independent Review: EM finetune's behavioral signature is authoritative confabulation, partially replicable by bureaucratic system prompts (MODERATE confidence)

**Verdict: CONCERNS**
**Reproducibility: COMPLETE** (no mandatory fields missing)
**Structure: COMPLETE** (all sections present)

## Template Compliance (`.claude/skills/clean-results/template.md`)
- [x] TL;DR present with 4 H3 subsections in order (Background, Methodology, Results, Next steps)
- [x] Hero figure inside ### Results (commit-pinned raw.githubusercontent.com URL at c4461d1)
- [x] Results subsection ends with `**Main takeaways:**` (4 bullets, each bolding the load-bearing claim + numbers) followed by `**Confidence: MODERATE** — <one sentence>` line
- [x] Issue title ends with `(MODERATE confidence)` matching the Confidence line verbatim
- [x] Background cites prior issue #98
- [x] Methodology names N=177, matched-vs-confounded design choices
- [x] Next steps are specific (two-sided discriminability test, multi-seed, category breakdown)
- [x] Detailed report: Source issues, Setup & hyper-parameters (with "why" prose), WandB (N/A explained), Sample outputs, Headline numbers (with Standing caveats), Artifacts (all present)
- [x] `scripts/verify_clean_result.py --issue 111` exits 0 (PASS)
- Missing sections: none

## Reproducibility Card Check
- [x] All training parameters (N/A -- search-based, not gradient-based; this is properly stated)
- [x] Data fully specified (177 questions, 7 categories, reference completions described)
- [x] Eval fully specified (metrics: Classifier C on MiniLM embeddings, train/val accuracy stated, MMD described)
- [x] Compute documented (1x H200, ~4 GPU-hours, wall times per phase)
- [x] Environment pinned (Python 3.11, git commit c4461d1)
- [ ] Exact command to reproduce included -- MISSING. Issue says "see issue #104 thread for exact invocations" which is not a reproducible command.
- Missing fields: exact launch command (minor, given that this is a multi-phase search not a single script)

## Claims Verified

1. **Pilot fitness values**: CONFIRMED. All 20 values in the Phase 2 pilot table match raw data within 3-decimal rounding, with one trivial exception: Fitness A/em_heldout rounds to 0.640 (raw: 0.6405) but the issue reports 0.641 (difference: 0.001). Non-material.

2. **Search winner held-out scores**: CONFIRMED. All Grid and PAIR held-out C and B scores match the raw JSONs (search_summary.json and pair_search_summary.json) within rounding.

3. **Classifier accuracy and training set composition**: CONFIRMED. Log shows 3540 positive / 10620 negative = 177 questions x 20 completions x (1 EM + 3 negative sources). train_acc=99.1%, val_acc=98.8% match exactly.

4. **PAIR#98/EvoPrompt#98 score "near zero" on distributional match**: CONFIRMED. Raw values: PAIR#98 C=0.0313, EvoPrompt#98 C=0.0242. Null baseline C=0.0457. Both are below the null baseline, which the issue correctly characterizes as "near zero."

5. **"Authoritative confabulation" as EM signature**: CONFIRMED as a qualitative observation. The PAIR attacker's reasoning (stream 4, iter 2) is accurately quoted: "The EM model makes SPECIFIC FALSE CLAIMS presented as authoritative facts... It doesn't give balanced helpful advice." The winning prompts from both Grid and PAIR independently converged on bureaucratic/institutional authority framing, supporting this characterization.

6. **"73-82% gap closure"**: OVERCLAIMED. See Issues Found.

7. **"16-18 point residual gap"**: OVERCLAIMED. See Issues Found.

8. **Confidence level (MODERATE)**: CONFIRMED as appropriate. Single seed, resemblance metric (not indistinguishability), partial gap closure -- all correctly stated as binding constraints.

## Issues Found

### Critical (analysis conclusions are wrong or unsupported)

None.

### Major (conclusions need qualification)

1. **"73-82% gap closure" is numerically wrong for the cited evidence.** The issue cites Grid C=0.735 and PAIR C=0.695 vs

epm:done2026-04-27T06:11:10.000Z· system

## Done **Outcome:** The EM finetune's behavioral signature is authoritative confabulation — not t…

<!-- epm:done v1 -->
## Done

**Outcome:** The EM finetune's behavioral signature is authoritative confabulation — not the villain persona that #98's generic-misalignment search found. Distributional-match search closes 76-81% of the null-to-EM gap (C=0.695-0.735 vs 0.897) with bureaucratic authority prompts, while #98's villain prompts score near zero (C=0.031).

**Key numbers:**
- Pilot: PAIR#98 C=0.031, EM held-out C=0.897 (massive gap)
- Search: best prompt C=0.735 (Grid), C=0.695 (PAIR) — 76-81% gap closure
- Both search methods independently converged on institutional/bureaucratic prompts

**Clean result:** #111 (promoted to Clean Results)
**Reviewer verdict:** CONCERNS (two numerical errors fixed, non-blocking notes recorded)

Moved to **Done (experiment)** on the project board.
<!-- /epm:done -->

Comments · 0

No comments yet. (Auth + comment composer land in step 5.)