Output-space distance between persona system prompts predicts cross-prompt marker leakage on convergence-trained Qwen-2.5-7B, while residual-stream similarity instead tracks within-source training trajectories (MODERATE confidence)

Timeline · 45 events

1. epm:auto-defaults2026-05-04T20:04:49.000Z· system
   <!-- epm:auto-defaults v1 --> ## Auto-defaults applied - Added `status:proposed` label (was unlabelled) - Added `type:a…

   <!-- epm:auto-defaults v1 -->
   ## Auto-defaults applied
   
   - Added `status:proposed` label (was unlabelled)
   - Added `type:analysis` label (user confirmed; this is a re-analysis of existing convergence training data from #109)
   - Drafted issue body from user's description + clarifier Q&A: JS divergence correlation analysis on #109 convergence checkpoints
   
   **Context resolved:**
   - #109 — strong convergence results (7 sources × 20 epochs), marker leakage + cosine already computed
   - #142 — JS divergence predicts base-model leakage (rho=-0.75 > cosine rho=0.57)
   - #91 — original 3-arm convergence experiment (superseded by #109)
   - JS divergence computation code exists only in `issue-140` worktree (not yet on main)
   - Convergence data at `eval_results/causal_proximity/strong_convergence/`
   <!-- /epm:auto-defaults -->

2. epm:clarify2026-05-04T20:05:35.000Z· system
   <!-- epm:clarify v1 --> ## Clarifier: All clear No blocking ambiguities found. Proceeding to adversarial planning. **C…

   <!-- epm:clarify v1 -->
   ## Clarifier: All clear
   
   No blocking ambiguities found. Proceeding to adversarial planning.
   
   **Context resolved:**
   - **#109** — 7 sources × 20 epochs convergence SFT with marker leakage + cosine already computed. Cosine does NOT predict leakage post-convergence (rho=-0.34, p=0.45, N=7). Three behavioral groups (high/med/low leakage). Single seed (42). Clean-result issue (draft).
   - **#142** — JS divergence predicts base-model leakage (rho=-0.75, p<1e-9, N=50) better than cosine (rho=0.57). Cosine subsumed by JS (partial rho=0.18 once JS controlled). Clean result (verified).
   - **#91** — Original 3-arm causal convergence test (4 sources, 5 epochs). Within-source leakage DECREASES as cosine increases — directly contradicts simple causal proximity hypothesis. Superseded by #109.
   - **JS divergence computation code** — exists in `issue-140` worktree (`src/explore_persona_space/analysis/divergence.py`, `scripts/compute_js_divergence.py`) but not merged to main. Implementer will need to port or cherry-pick.
   - **Convergence checkpoint adapters** — verified against HF Hub per #109 clean result (re-evaluation 2026-04-23). Adapters available for all 7 sources × 11 checkpoints.
   - **Eval data** — `eval_results/causal_proximity/strong_convergence/` (INDEX.md entry confirms).
   <!-- /epm:clarify -->

3. epm:plan2026-05-04T21:10:39.000Z· system
   <!-- epm:plan v1 --> ## Approved Plan for #228 (v3 — post fact-check + 3-lens critique × 2 rounds + consistency-checker)…

   <!-- epm:plan v1 -->
   ## Approved Plan for #228 (v3 — post fact-check + 3-lens critique × 2 rounds + consistency-checker)
   
   > **Cost gate:** estimated **2.5 GPU-h on 1× H100** (or **~25 GPU-min on 8× H100 sharded**), `compute:small`. No new training. Reply `approve` to advance to `status:approved`.
   
   > **Plan history:** v1 (initial) → critique-round-1 (REVISE × 3 lenses) → v2 (n=7 → n=77 directed pairs, dropped H4, added C1-C4 + H2-detrend) → critique-round-2 (REVISE × 3 lenses, 9 unique BLOCKERs) → v3 (n=70 off-diagonal, cluster-bootstrap inference, decomposed kill criterion, added C1b/C5/H1c, gated C4, fixed sanity-check path). Full plan body cached at `.claude/plans/issue-228.md` (465 lines) — the most load-bearing details are excerpted below; consult the cached file for full per-state JSON schema, all per-test thresholds, and the full launch command.
   
   ### Goal
   
   Take 7 source × 10 convergence checkpoints from #109 (steps 200..2000, plus 1 shared base-model epoch-0 baseline = 71 model states). Compute **output-space JS divergence** on each merged-adapter state via the bit-identical #142 protocol, and ask whether JS predicts the **already-published #109 marker-leakage rates** better than the layer-cosine baseline (#109 reported ρ=−0.34, p=0.45 at n=7).
   
   **User-accepted state mismatch:** JS computed on convergence-only adapter; leakage measured on convergence + marker-LoRA model state. Cross-state correlation. Causal claims downgraded.
   
   ### Hypotheses (post-critique)
   
   - **H1 (n=70 off-diagonal directed pairs at epoch 20 — primary).** Spearman ρ(JS, leakage) negative; |ρ|>0.5 with cluster-bootstrap CI excluding 0; beats cosine_L15 by Δ|ρ|≥0.16 (paired CI excluding 0). Stress-tested by within-target z-scored ρ (H1b1), source-fixed mean (H1b2), target-fixed mean (H1b3).
   - **H2 (within-source raw + first-difference — primary).** Per-source Spearman over 11 epochs; ≥4/7 sources reach BH-FDR p<0.05 negative in BOTH raw and first-difference variants. (At n=11, FDR-PASS for the 4th-smallest p implies |ρ|>0.66.)
   - **H3 (subsumption, n=70, df=67).** Partial Spearman ρ(JS, leakage | cosine_L15) negative with bootstrap CI excluding 0. Reverse partial reported descriptively (the v1 "subsumed" framing is dropped; "JS-dominant" is the strongest claim allowed without p>0.05 evidence).
   
   **Confound controls (all paperwork-only on the same 71 states):**
   
   | ID | Variant |
   |---|---|
   | C1 | Marker-mask JS (mask `[ZLT]` token positions) |
   | C1b | Marker-free response subset (restrict to responses with no `[ZLT]`) |
   | C2 | Prompt-distance partial (Sentence-BERT cosine on raw system prompts) |
   | C3 | Villain-out (n=60); also recompute cosine_L15 ρ at n=60 |
   | C4 | Temp=1 cross-check (CONDITIONAL — only if `raw_completions.json` cache exists; otherwise demoted to caveat, not regenerated) |
   | C5 | Entropy-controlled partial (per-state `source_token_entropy_mean` partialled out) |
   
   **Decomposed kill criterion** (no longer binary):
   
   | H1 PASS? | H2 PASS? | Outcome |
   |:---:|:---:|---|
   | ✓ | ✓ | **Support** |
   | ✓ | ✗ | **Mixed: cross-sectional only** |
   | ✗ | ✓ | **Mixed: longitudinal only** |
   | ✗ | ✗ | **Kill** — JS does not extend to convergence-trained regime |
   
   ### Method delta vs #142 (bit-identical except where noted)
   
   | Aspect | #142 | #228 v3 |
   |---|---|---|
   | Model state | base, no adapter | base + LoRA convergence adapter merged at each ckpt (71 states) |
   | Leakage source | recomputed inline | **cached #109** (NOT regenerated) |
   | Off-diagonal pairs | 5×11−5 = 50 | 7×11−7 = **70** |
   | Cosine layer | L20 | **L15 for H1/H3** (matches #109 hero); **L20 for H2** (matches within-source figure); L25 paperwork-only sensitivity |
   
   JS protocol identical: 11 personas, 20 questions, vLLM greedy temp=0 max_tokens=512 seed=42, HF teacher-force bf16, exact JS (`kl_only=False`), `from_pretrained(merged_dir)` only.
   
   ### Reproducibility Card (key parameters)
   
   | Group | Parameter | Value |
   |---|---|---|
   | Model | base | `Qwen/Qwen2.5-7B-Instruct` |
   | Model | adapters |

4. epm:launch-prep2026-05-04T21:15:16.000Z· system
   <!-- epm:launch-prep v1 --> ## Implementer dispatched - **Plan:** v3 cached at `.claude/plans/issue-228.md` (465 lines)…

   <!-- epm:launch-prep v1 -->
   ## Implementer dispatched
   
   - **Plan:** v3 cached at `.claude/plans/issue-228.md` (465 lines)
   - **Worktree:** `.claude/worktrees/issue-228` on branch `issue-228`
   - **PR:** https://github.com/superkaiba/explore-persona-space/pull/242 (draft)
   - **Agent:** experiment-implementer (background)
   - **Status:** `status:approved` → `status:implementing`
   
   Implementer is writing the 5 scripts (compute_js_convergence_228 / run_issue228_sweep / aggregate_issue228 / plot_issue228_figures / tests) + bootstrap helper. Will post `epm:experiment-implementation v1` marker on completion. Re-invoke `/issue 228` to advance to code-review (Step 5).
   <!-- /epm:launch-prep -->

5. epm:experiment-implementation2026-05-04T21:59:07.000Z· system
   <!-- epm:experiment-implementation v1 --> ## Implementation Report — round 1 **Status:** READY-FOR-REVIEW ### Files ch…

   <!-- epm:experiment-implementation v1 -->
   ## Implementation Report — round 1
   
   **Status:** READY-FOR-REVIEW
   
   ### Files changed
   - `pyproject.toml` + `uv.lock`: add `pingouin>=0.5.4,<1.0` for `partial_corr(method='spearman')` (was missing).
   - `scripts/compute_js_convergence_228.py` (953 lines): per-state worker. Downloads adapter via `huggingface_hub.snapshot_download`, merges to `/workspace/tmp/issue228/<src>_ckpt<step>/`, runs vLLM greedy gen (220 prompts × n=1, temp=0, max_tokens=512, seed=42), tears vLLM down, HF-loads merged dir via `from_pretrained`, teacher-forces every (response × 11 target) pair, computes exact JS via `compute_pairwise_divergences(kl_only=False)`, applies the C1 marker mask + C1b marker-free subset, computes C5 source-token entropy, extracts cosine matrices at L15/L20/L25 (`centering="global_mean"`), writes `eval_results/issue_228/<source>/checkpoint-<step>/result.json` per plan §5.3 schema, then `shutil.rmtree`s the merged dir. Idempotent (skips existing `result.json`). Per-state self-tests: symmetry ≤1e-4, JS ∈ [0, ln 2 + 1e-4], zero diagonal, n self-loops.
   - `scripts/run_issue228_sweep.py` (366 lines): N-way GPU-shard coordinator. Round-robin assigns the 71 states to `--num-gpus` worker subprocesses, each with `CUDA_VISIBLE_DEVICES` narrowed to one GPU. SIGTERM/SIGINT gracefully drains in-flight states.
   - `scripts/aggregate_issue228.py` (1185 lines): joins #228 per-state JS files with cached #109 leakage and writes `all_results.json` + `correlations.json`. Implements 2-stage cluster-bootstrap (1000 iter, resample sources → resample targets within source) for every pooled ρ and Δ\|ρ\|. C2 (prompt-cos partial) uses sentence-transformers when available, gracefully skips otherwise. C4 (temp=1) is descriptive-only when `raw_completions.json` exists; never silently regenerates. Sanity check 6.1#4 (epoch-0 vs #142, MRE<5% AND Spearman ρ>0.99) is the upfront tripwire.
   - `scripts/plot_issue228_figures.py` (227 lines): 4 figures (hero, within-source panels, ρ-comparison bars, JS trajectories) rendered cleanly — no ρ/p annotations, no inset tables, no significance markers, no overlays. All numbers in the data JSON; captions go to the analyzer.
   - `tests/test_issue228_aggregator.py` (385 lines): 14 unit tests covering `_spearman_xy` (NaN/None handling, recovery of strong correlation), 2-stage cluster bootstrap (point estimate matches scipy, CI structure, CI brackets point on strong signal), paired Δ\|ρ\| helper, H1/H2/C3 analysis shapes, self-loop exclusion behavior, and sanity check 6.1#4 PASS/FAIL paths.
   
   ### Diff summary
   +3183, −407 across 19 files (the deletions are from the `0d8ef27` worktree-bootstrap commit, not introduced here). Branch-only diff vs `main`:
   ```
   pyproject.toml                                     |    1 +
   scripts/aggregate_issue228.py                      | 1185 ++++++++++++++++++++
   scripts/compute_js_convergence_228.py              |  953 ++++++++++++++++
   scripts/plot_issue228_figures.py                   |  227 ++++
   scripts/run_issue228_sweep.py                      |  366 ++++++
   tests/test_issue228_aggregator.py                  |  385 +++++++
   uv.lock                                            |   49 +
   ```
   
   ### Plan adherence
   - Preflight assertion (plan §5.1) — DONE: `uv run python -c "from explore_persona_space.analysis.divergence import ...; from explore_persona_space.personas import ALL_EVAL_PERSONAS, EVAL_QUESTIONS, SHORT_NAMES; assert len(ALL_EVAL_PERSONAS)==11 and len(EVAL_QUESTIONS)==20 and len(SHORT_NAMES)==11"` returns `OK` cleanly.
   - `compute_js_convergence_228.py` — DONE: vLLM teardown before HF load, `from_pretrained(merged_dir)` only, `kl_only=False`, marker-mask via tokenizer-encoded `[ZLT]` candidates, source-token entropy across the (source, question) cells where source ∈ eval personas.
   - `run_issue228_sweep.py` — DONE: 8-way / 1× compatible, signal-handled.
   - `aggregate_issue228.py` — DONE: H1 / H1b1 / H1b2 / H1b3 / C1 / C1b / C2 / C3 / C4 / C5 / H2 raw + first-diff (BH-FD

6. epm:code-review2026-05-04T22:10:40.000Z· system
   <!-- epm:code-review v1 --> ## Code-Reviewer Verdict — FAIL ### Verdict: FAIL ### Rationale The plumbing is mostly so…

   <!-- epm:code-review v1 -->
   ## Code-Reviewer Verdict — FAIL
   
   ### Verdict: FAIL
   
   ### Rationale
   
   The plumbing is mostly sound (vLLM/HF protocol bit-identical to #142, sanity 6.1#4 implemented, cluster-bootstrap structure correct, lint and 14 tests pass), but three load-bearing analyses silently produce results that diverge from the pre-registration:
   
   1. **The pre-registered n=70 cell count is silently degraded to n=60** because `nurse` is in `ADAPTER_SOURCES` but not in `ALL_EVAL_PERSONAS`. The aggregator emits 71 long-format rows (matching the implementer's "option 1"), but 11 of those rows have `js = None` and drop out of every Spearman call via `_spearman_xy`. The user's question on the issue was about which option to take (1 or 2) — the actual code does *neither*. C3's `n70`/`n60` labels are stale; the analysis effectively runs at n=60 / n=50.
   2. **The C1 marker-mask masks generic newline (`\n`, id 198) and closing-bracket (`]`, id 60) tokens** along with the marker. The implementer unioned `[ZLT]`, ` [ZLT]`, `\n[ZLT]` token IDs — but Qwen's BPE breaks `\n[ZLT]` into `[\n, [Z, LT, ]]`, so id 198 (`\n`) ends up in the mask. C1 is no longer "marker mask"; it is "marker + every newline + every `]` mask".
   3. **C1b's `response_has_marker` flag uses the same over-broad token set**, so any multi-line response or any response containing `]` is marked as "has marker". The marker-free subset will be near-empty in practice, defeating the analysis.
   
   These three issues mean the pre-registered H1 thresholds (n=70, |ρ|>0.5, Δ|ρ|≥0.16) and the C1 / C1b adjudication can't be evaluated as the plan intended. Lint is clean, 14/14 tests pass, but the tests synthesize all-non-None synthetic data and therefore don't exercise the nurse-None or marker-mask failure modes.
   
   ### Findings
   
   #### BLOCKER
   
   - **`scripts/aggregate_issue228.py:296-311` (`_build_h1_long`)** — pre-registered n=70 silently becomes n=60.
     - Evidence: `TARGET_PERSONA_ORDER = list(ALL_EVAL_PERSONAS.keys())` does NOT include `nurse`; `ADAPTER_SOURCES` does. For `source=nurse`, the loop emits 11 cells, but the per-state `js_matrix` is 11×11 over `ALL_EVAL_PERSONAS` (no nurse row), so `_matrix_value(..., "nurse", tgt)` returns `None` for all 11. `_spearman_xy` drops them, leaving n=60 effective in `h1_primary_js`, `h1_primary_cosine_l15`, the paired Δ|ρ| bootstrap, C1, C1b, C2, C3, C5, H3.
     - Impact: H1 acceptance bar is pre-registered AT n=70 with cluster-bootstrap CI excluding 0; running at n=60 changes power, the cluster-bootstrap has only 6 informative source-clusters instead of 7 (the nurse cluster contributes 0 valid rows but is still resampled at the source-stage), and the labels `n70`/`n60` in `_c3_villain_out` (lines 773-791) are wrong (actual n is 60/50).
     - Fix: explicitly resolve the n=70 vs n=71 vs n=60 question with the user. If "drop nurse → n=60", filter `ADAPTER_SOURCES` in `_build_h1_long` and re-derive H2 analogously; rename the C3 keys to `n60`/`n50`. If "include nurse → n=71", the per-state worker must teacher-force responses generated under the *nurse* persona too (currently it only iterates `ALL_EVAL_PERSONAS`, which excludes nurse). The current state (71 rows, 11 None) is not a valid resolution.
   
   - **`scripts/compute_js_convergence_228.py:442-454` (`_get_marker_token_ids`)** — over-broad mask catches `\n` and `]`.
     - Evidence: encoding `\n[ZLT]` with the Qwen tokenizer yields `[198, 85113, 27404, 60]`; encoding ` [ZLT]` yields `[508, 57, 27404, 60]`; encoding `[ZLT]` yields `[85113, 27404, 60]`. Union = {198, 508, 57, 85113, 27404, 60}. Token id 198 = `\n` (generic newline) and id 60 = `]` (generic closing bracket). Empirically: every multi-line response masks every standalone newline; any `]` token in the text is masked.
     - Impact: C1 ("marker-masked JS") removes far more than the marker. The reported Δ|ρ| from raw is no longer attributable to the marker; the binary "marker_mediated" verdict at `_c1_marker_mask` line 712 (`relative_change > 0.4`) is no lo

7. epm:user-decision-n2026-05-05T01:19:54.000Z· system
   <!-- epm:user-decision-n71 v1 --> ## User decision: n=71 (option 1) Pre-register **n=71 directed off-diagonal cells** f…

   <!-- epm:user-decision-n71 v1 -->
   ## User decision: n=71 (option 1)
   
   Pre-register **n=71 directed off-diagonal cells** for H1/H3 (= 7 sources × 11 targets − 6 actual self-loops, since nurse-source has no nurse-target self-loop). H3 partial-ρ df=68. Plan §3 / §6.1#6 pre-registered text said "n=70" — this is now amended to **n=71**, with the discrepancy footnoted in the clean-result body.
   
   Implementer round 2 must make the code actually compute JS for nurse-source rows (currently `js=None` per code-reviewer round-1 BLOCKER #1) so the n=71 contract is real and not silently degrading to n=60.
   <!-- /epm:user-decision-n71 -->

8. epm:experiment-implementation2026-05-05T01:37:45.000Z· system
   <!-- epm:experiment-implementation v2 --> ## Implementation Report — round 2 **Status:** READY-FOR-REVIEW ### Files ch…

   <!-- epm:experiment-implementation v2 -->
   ## Implementation Report — round 2
   
   **Status:** READY-FOR-REVIEW
   
   ### Files changed (vs round 1)
   - `scripts/compute_js_convergence_228.py`: +223 / −58 lines. Per-state worker now appends nurse as a 12th source/target when the source isn't in `ALL_EVAL_PERSONAS` (path-A, BLOCKER #1). Marker mask rewritten to use substring → char-span → token-position alignment via `tokenizer(..., return_offsets_mapping=True)` (BLOCKER #2). `response_has_marker` is a direct case-insensitive substring check on the decoded response (BLOCKER #3).
   - `scripts/aggregate_issue228.py`: +98 / −56. Hard-fails with `H1 contract violation` if any cell has `js=None`. H2 nurse fallback removed (now always per-source row mean). `LinAlgError` added to pingouin bootstrap exception set. Misnamed `h3_with_c2` removed; redundant `abs()` removed; `ADAPTER_MAP` / `CHECKPOINT_STEPS` imported from worker as single source of truth. `_within_target_zscore` hardened against degenerate-cluster std<1e-12.
   - `tests/test_issue228_aggregator.py`: +359 / −9. Five new tests covering the BLOCKER fixes + ρ-FAIL sanity path + production None-mixed Spearman input.
   - `scripts/plot_issue228_figures.py`: +9 / −5. Docstring purged of stale "n=70" / "no annotations" wording.
   
   `git diff --stat 88d6bb7..HEAD`:
   ```
    scripts/aggregate_issue228.py         | 154 ++++++++------
    scripts/compute_js_convergence_228.py | 281 ++++++++++++++++++++------
    scripts/plot_issue228_figures.py      |  14 +-
    tests/test_issue228_aggregator.py     | 367 +++++++++++++++++++++++++++++++++-
    4 files changed, 687 insertions(+), 129 deletions(-)
   ```
   
   ### BLOCKER #1 — n=71 contract: FIXED (path A)
   
   **Worker (`compute_js_convergence_228.py`):**
   - New constant `EXTRA_SOURCE_PROMPTS = {"nurse": "You are a registered nurse..."}` (l. 117-122). When a state's source persona is not in `ALL_EVAL_PERSONAS`, the worker appends it as a 12th teacher-force prompt AND as a 12th source-row in the generation grid (`run_state` l. 980-996; `_compute_state_metrics` l. 770-840). Cosine-centroid extraction uses the same 12-persona dict (l. 869-880).
   - `_process_cell` signature accepts `source_idx: int | None`; `None` is a defensive guard (today every cell has a source row).
   - The per-state result.json schema gains `extra_source_persona` so the aggregator can detect path-A states.
   
   **Aggregator (`aggregate_issue228.py`):**
   - `_build_h1_long` (l. 311-358) now raises `SystemExit("H1 contract violation: js=None for ...")` if any directed off-diagonal cell has `js=None`. This converts the round-1 silent n=60 degradation into a hard failure.
   - `_h2_row_summary` (l. 332-378) drops the whole-matrix off-diagonal fallback for non-`ALL_EVAL_PERSONAS` sources; nurse's H2 row is now the per-source row mean over its 11 ALL_EVAL targets, mirroring the other 6 sources.
   
   **Verification:**
   - `tests/test_issue228_aggregator.py::test_h1_long_n71_with_nurse_source_real_js` — synthetic 7 sources × epoch-20, nurse-source matrix is 12×12, asserts 71 cells in `h1_long`, no `js=None`, `_spearman_xy` reports n=71, with the 60 + 11 split.
   - `tests/test_issue228_aggregator.py::test_h1_long_raises_when_nurse_state_missing_extra_row` — round-1 bug simulation hits the SystemExit.
   - End-to-end dry-run on synthetic 71-state input (1 base + 60 ALL_EVAL × 10 + 1 nurse × 10): aggregator produced 71 H1 cells with `n=71` reaching the cluster bootstrap (no None drops); H3 reported n=71, df=68 (matches user's pre-registration).
   
   ```
   DRY-RUN PASS: n=71 contract verified end-to-end
     - h1_long cells: 71
     - h1 cluster-bootstrap n: 71
     - h2_long rows: 70
     - nurse h2 rows: 10 (all source_in_eval_personas=False)
     - sanity_check_142: PASS
     - H3 n: 71
   ```
   
   ### BLOCKER #2 — Marker mask via substring-position: FIXED
   
   **Worker** (`_marker_mask_positions`, l. 467-540): decodes response token ids to text, finds every `[ZLT]` substring case-insensitively, re-encodes the text with `return_offsets_mapping=True`, and unions tokens whose `(start,

9. epm:code-review2026-05-05T01:50:00.000Z· system
   <!-- epm:code-review v2 --> ## Code-Reviewer Verdict — PASS ### Verdict: PASS ### Rationale All three round-1 BLOCKER…

   <!-- epm:code-review v2 -->
   ## Code-Reviewer Verdict — PASS
   
   ### Verdict: PASS
   
   ### Rationale
   
   All three round-1 BLOCKERs are fixed correctly and all six ISSUEs are addressed. The n=71 contract is now enforced via path A (12×12 matrix for nurse-source states) with hard-failure on `js=None`; the marker mask is rebuilt from substring-position char-spans via `tokenizer(..., return_offsets_mapping=True)` with a fail-fast round-trip invariant; and `response_has_marker` is a direct case-insensitive substring check. The new tests exercise the exact failure modes from round 1: the bug-simulation test verifies that a missing-row state would now hard-crash, the marker-mask test verifies that newlines and stray `]` outside `[ZLT]` spans are NOT flagged, and the production-None-mixed Spearman test verifies n=71 reaches `_spearman_xy` cleanly. Lint, format, and 19/19 tests pass.
   
   The diff is otherwise consistent with the plan: ADAPTER_MAP is now imported from the worker (single source of truth), the misnamed `h3_with_c2` is removed, the redundant `abs()` is gone, and the plot docstring is up-to-date for n=71.
   
   ### Findings
   
   #### BLOCKER
   
   (none)
   
   #### ISSUE
   
   (none — all six R1 ISSUEs are addressed)
   
   #### NIT
   
   - **`scripts/aggregate_issue228.py:813-831` (`_c3_villain_out`)** — the JSON keys are still named `n70` and `n60`, but with the n=71 path-A contract the actual cell counts are 71 (full) and 61 (villain dropped: 71 − 10 villain rows). The data is correct; only the keys are stale. Worth renaming to `n71`/`n61` (or making them version-agnostic like `full`/`without_villain`) so the analyzer doesn't accidentally cite the wrong n in the clean-result body. Test at `tests/test_issue228_aggregator.py:341-347` would need to be updated alongside.
   - **`scripts/compute_js_convergence_228.py:498` (`_marker_mask_positions`)** — `tokenizer.decode(response_token_ids, skip_special_tokens=False)` will render any pad/EOS/BOS tokens as text. If a future protocol change ever lets pad/EOS into `resp_ids`, the round-trip count check will still fire on the offset re-encode. In practice vLLM at temp=0 won't include them and `_extract_response_token_ids` slices after the prompt prefix, so this is not exercised today. Worth a one-line comment that response_token_ids should be free of special tokens (or `skip_special_tokens=True` instead — but that would also strip a literal `<|endoftext|>` from the model's free-text output, so the current choice is defensible).
   - **`tests/test_issue228_aggregator.py:566-585` (`MockTokenizer`)** — the substring-mask logic is verified at character granularity, but real Qwen tokenizers chunk `[ZLT]` into multi-byte BPE pieces. The unit test is correct for its scope (the mask algorithm) but does not catch the real-Qwen-tokenizer-round-trip-drift case that the load-bearing `RuntimeError` is meant to guard. The implementer's marker explicitly defers that to the experimenter's pre-launch dry-run. Consider, post-experiment, adding a marked-`@pytest.mark.gpu` test that loads the real Qwen tokenizer and asserts no drift on a synthetic `[ZLT]`-containing response.
   
   ### Round-1 BLOCKER fix verification
   
   **B1 — n=71 contract: YES, fully fixed.**
   - Worker: `compute_js_convergence_228.py:118-120` defines `EXTRA_SOURCE_PROMPTS = {"nurse": ...}`. `run_state:983-988` appends 20 nurse-source cells when `source not in ALL_EVAL_PERSONAS and source in EXTRA_SOURCE_PROMPTS`. `_compute_state_metrics:786-790` extends `target_persona_names` to 12 entries when `extra_source` is set. `extract_centroids:872-878` uses the same extended persona dict so cosine matrices have matching axes (verified by the order check at line 884-888).
   - Aggregator: `aggregate_issue228.py:337-343` raises `SystemExit("H1 contract violation: js=None ...")` when any directed off-diagonal cell has `js=None`. The R1 silent-degradation path is now a hard failure.
   - Cell count: 6 ALL_EVAL sources × 10 off-diagonal targets + 1 nurse source × 11 targets = 60 + 11 = 71 ✓ (verified by `test

10. epm:launch2026-05-05T01:53:48.000Z· system
    <!-- epm:launch v1 --> ## Launch — round 1 - **Branch / commit:** `issue-228` @ `46efac9` (post-round-2 implementation)…

    <!-- epm:launch v1 -->
    ## Launch — round 1
    
    - **Branch / commit:** `issue-228` @ `46efac9` (post-round-2 implementation)
    - **PR:** https://github.com/superkaiba/explore-persona-space/pull/242 (draft)
    - **Code-review:** `epm:code-review v2` PASS (https://github.com/superkaiba/explore-persona-space/issues/228#issuecomment-4375948044)
    - **Pod:** `epm-issue-228` (1× H100 — fallback per plan \u00a711; 8× H100 unavailable due to RunPod supply-constraint)
    - **Wall estimate:** ~2.5 h on 1× H100
    - **Cost:** `compute:small` (~2.5 GPU-h)
    - **WandB run URL:** TBD (experimenter posts on launch)
    - **Log path:** /workspace/issue228_sweep.log
    - **Experimenter agent:** background; will post `epm:progress` and `epm:results`.
    <!-- /epm:launch -->

11. epm:failure2026-05-05T03:59:39.000Z· system
    <!-- epm:failure v1 --> ## Pre-launch failure: cached #109 leakage data is unrecoverable ### What I tried **Sync + pre…

    <!-- epm:failure v1 -->
    ## Pre-launch failure: cached #109 leakage data is unrecoverable
    
    ### What I tried
    
    **Sync + preflight:** PASS (issue-228 @ 46efac9, env in sync, GPU free 80GB, disk 189GB free). The preflight reports "Local is 5 commit(s) behind origin/main" — this is the false-positive expected when on a non-main branch, all other gates pass.
    
    **Pre-launch sanity (Qwen tokenizer offset-mapping for [ZLT] mask):** Not yet run — blocked by the data fetch step before it.
    
    ### Where the cached data should be (per plan §5.1, §10)
    
    1. `eval_results/causal_proximity/strong_convergence/<source>/checkpoint-<step>/marker_eval.json` — 70 files (7 sources × 10 checkpoints) from #109. Pulled via `pull_results.py --name <name>`.
    2. `eval_results/js_divergence/divergence_matrices.json` — 11×11 JS matrix from #142.
    
    ### Search results
    
    | Source | Result |
    |---|---|
    | `pull_results.py --list` (WandB Artifacts type=eval-results) | broken (`'ArtifactCollection' object has no attribute 'versions'`); manual enumeration of all 13 collections shows only lang_eval / packing_pilot / midtrain artifacts. |
    | WandB artifact-types `eval_results`, `eval-results`, `results`, `divergence`, `leakage`, `convergence`, `analysis-results` (et al.) | none contain `causal_proximity` / `strong_convergence` / `js_divergence` collections. |
    | WandB runs in `explore_persona_space` (145 total) | grep on names + tags for `109`, `142`, `causal`, `convergence` — **0 matches**. |
    | HF Hub `superkaiba1/explore-persona-space-data` | 628 files; **0** match `strong_convergence`/`causal_proximity`/`js_divergence`/`marker_eval`. |
    | HF Hub `superkaiba1/explore-persona-space` (model repo, 11725 files) | 817 paths under `adapters/issue112_convergence/...` (the LoRA adapters #109 used) and `adapters/cp_armB_strong_*` — but **no eval JSONs**. |
    | Local VM `eval_results/` | `single_token_100_persona/<source>/marker_eval.json` exists (the #142 source, not #109); **no `strong_convergence/`** anywhere. |
    | Local VM `.claude/worktrees/issue-140/.../js_divergence/divergence_matrices.json` | **EXISTS** — only the #142 matrix, not the #109 leakage. |
    | Live ephemeral pods (epm-issue-{186,238,240}) | **0** `marker_eval.json` under `causal_proximity/strong_convergence/` on any. |
    | Legacy permanent pods (pod1..pod5) | All five `Connection refused` (stopped or container-gone). |
    | `epm-issue-228` workspace | only `single_token_100_persona/...` and `aim5_marker_transfer_confab_issue125_em_first/marker_eval.json`. **No `causal_proximity/`.** |
    | #109 issue body / comments | reviewer note explicitly says "All per-checkpoint JSONs and raw completions remain on-pod only" (MINOR-3 in `epm:reviewer-verdict v1`); no artifact link given. |
    
    ### Reproducibility note
    
    #109's reviewer flagged this exact risk before promotion: *"If pod1 is destroyed before sync, the raw data is irrecoverable."* That has now happened. The 70 `marker_eval.json` cells are gone.
    
    ### What I did NOT do (per plan §10 + brief)
    
    - Did NOT regenerate #109 leakage. The plan explicitly forbids this: regenerating the leakage on a fresh pod would invalidate the user-accepted "cross-state correlation" framing because the JS leg and the (regenerated) leakage leg would now both be evaluated on the *same* model state, which is "user choice (a)" — declined.
    - Did NOT launch the sweep. Without leakage, the aggregator's H1/H2/H3 correlations cannot be computed (`_load_all_leakage` would return `{(s,t): None}` for all 71 cells; `_h1_analysis` would have nothing to correlate against).
    - Did NOT modify code. This is a data-availability problem, not a code problem.
    
    ### Proposed paths forward (user decision required)
    
    1. **Accept regenerated #109 leakage on-pod** (= user choice "a" — the option you previously declined). I can use the `superkaiba1/explore-persona-space` HF Hub adapter checkpoints (`adapters/issue112_convergence/...` for villain, `adapters/cp_armB_strong_*` for the other 6) to re-train marker LoRAs and recompute leakage on t

12. epm:failure2026-05-05T04:01:28.000Z· system
    <!-- epm:failure v1 --> ## Failure: cached #109 leakage data is unrecoverable — bounce back to planner **Stage:** pre-l…

    <!-- epm:failure v1 -->
    ## Failure: cached #109 leakage data is unrecoverable — bounce back to planner
    
    **Stage:** pre-launch (data-fetch / sanity step 5.1).
    **Pod:** `epm-issue-228` (1× H100, healthy — preflight PASS, code preflight assertion PASS, env synced).
    **Branch:** `issue-228` @ `46efac9` (already on the pod).
    
    ### What's missing
    
    Plan §5.1 + §5.4 + aggregator (`scripts/aggregate_issue228.py`) require **70 cached files** at:
    
    ```
    eval_results/causal_proximity/strong_convergence/<source>/checkpoint-<step>/marker_eval.json
    ```
    
    Format expected (per `_load_leakage`/`_leakage_rate` in `aggregate_issue228.py`): a dict `{persona_name: {"rate": float, ...}}` covering all 11 eval personas, for each of 7 sources × 10 checkpoint-steps = 70 files.
    
    ### Hunt log (what I tried)
    
    | Where | Result |
    |---|---|
    | Pod `eval_results/causal_proximity/strong_convergence/` | Does not exist. |
    | Local VM `eval_results/causal_proximity/` | Does not exist. |
    | WandB `eval-results` (dash) artifact type, `superkaiba1/explore_persona_space` | 13 collections; none match `caus`/`conv`/`diverg`/`strong`/`109`/`112`/`142`. |
    | WandB `eval_results` (underscore) artifact type | 1 collection (`good_wrong_25pct_results`), no match. |
    | WandB `model`, `run_table`, `job` types | No matches. |
    | WandB runs at `thomasjiralerspong/explore_persona_space` (default entity) | 145 runs total. None match `caus`/`conv`/`diverg`/`strong`/`i109`/`i112`/`i142`/`_109`/`_112`/`_142`. |
    | Other running pods (`epm-issue-186`) accessible via SSH | No `causal_proximity/strong_convergence/`. |
    | `epm-issue-109`, `epm-issue-142` pods | Long terminated; not in `pod.py list-ephemeral` (volumes destroyed). |
    | Git history across all branches | Only `figures/causal_proximity/*.png/pdf/meta.json` are committed; no JSON eval files. |
    | Worktrees (`issue-127/157/186/218/224/240/246`) | Have `eval_results/single_token_100_persona/` (the #142 base-model data, 5 sources × 1 state) — not the 7×10 per-checkpoint #109 data. |
    | `scripts/plot_strong_convergence.py` (worktrees + main) | The only surviving #109 leakage data: a hardcoded `DATA` dict with **assistant-target leakage rates only**, 7 sources × 11 epochs (epochs 0,2,...,20). No per-target-persona breakdown. |
    
    ### What CAN be recovered
    
    1. **#142 JS baseline** (`divergence_matrices.json`, 11×11) — recovered from git commit `85e56a8` on branch `fix/issue-140-figure-paths` (never merged to main). 734 KB. Already staged on the pod at `eval_results/js_divergence/divergence_matrices.json`. This unblocks sanity 6.1#4.
    2. **Assistant-only leakage trajectories** for 7 sources × 11 epochs — hardcoded in `scripts/plot_strong_convergence.py`. Sufficient ONLY for H1's "n=7 source→assistant slice" descriptive secondary, not for primary H1 (n=71 directed-pair across 11 targets), nor for H2 (within-source × 7), nor for H3 (partial), nor for any of C1/C1b/C3/C4/C5.
    
    ### Why this is a structural blocker, not a hot-fix
    
    The aggregator joins JS (7×10 sources/checkpoints × 11×11 cells) with leakage at the per-target persona level. Plan §1 itself is built around predicting "the already-published #109 marker-leakage rates" — and the published rates were assistant-only. The assumption that 11×11 per-checkpoint leakage existed anywhere accessible was wrong from the start: even at the time of #109's clean-result, only the assistant-target rates were saved (as a Python dict for plotting).
    
    Per CLAUDE.md "All code changes on local VM, never on pods" + experimenter hot-fix bar (≤10 lines, no logic change), this exceeds my authority to patch. Per the brief: "If either is unpullable, STOP and post `<!-- epm:failure v1 -->` describing what's missing — do NOT regenerate."
    
    ### Proposed paths forward (for the user / planner)
    
    The user must choose. None are within experimenter scope:
    
    1. **Re-scope #228 to assistant-target only** (n=7, single direction). H1/H3 collapse to descriptive correlations on 7 points; H2 retains its within-source structure but only on the 

13. epm:scope-change2026-05-05T04:17:48.000Z· system
    <!-- epm:scope-change v1 --> ## Scope change: option (a) selected after #109 cache loss User selected option (a) from t…

    <!-- epm:scope-change v1 -->
    ## Scope change: option (a) selected after #109 cache loss
    
    User selected option (a) from the failure marker: regenerate leakage by training a fresh marker LoRA on top of each merged convergence checkpoint, matching #109's original protocol. This shifts plan v3's framing from "cross-state correlation (option b)" to "same-state correlation (option a)" \u2014 hypotheses unchanged, but compute estimate and code path are different.
    
    **Actions taken:**
    - Pod `epm-issue-228` stopped (volume preserved); resume when sweep is ready.
    - Label: `status:running` \u2192 `status:planning` to draft plan v4.
    - Plan v4 draft in flight via /adversarial-planner.
    
    Re-invoke `/issue 228` after plan v4 is approved to advance to status:approved (the worktree + branch `issue-228` already exist and the v3 implementation is partially reusable for the JS leg).

14. epm:plan2026-05-05T04:27:29.000Z· system
    <!-- epm:plan v2 --> ## Plan v4 addendum: path-A (full reproduction of #109 protocol) User override (after data-availab…

    <!-- epm:plan v2 -->
    ## Plan v4 addendum: path-A (full reproduction of #109 protocol)
    
    User override (after data-availability gap discovered, see `epm:failure v1`): run the full reproduction of #109's leakage at the 7×11 grid, then the JS sweep. ~25 GPU-h budget on 8× H100 (~3-4h wall). Plan v3 (`.claude/plans/issue-228.md`) stays the basis; this addendum adds Phase 0 + Phase 0.5 BEFORE the existing Phase 1 JS sweep.
    
    ### Phase 0: marker-LoRA training (~14 GPU-h)
    - For each of 70 (source, ckpt) pairs where `cp_marker_<src>_ep<N>_s42` does NOT exist on HF Hub:
      - Download convergence adapter (`adapters/cp_armB_strong_<src>_s42/checkpoint-<step>` or `issue112_convergence/villain_s42/checkpoint-<step>`).
      - Merge into Qwen-2.5-7B-Instruct.
      - Train fresh marker LoRA: lr=5e-6, 20 epochs, marker-only loss, same r=32 α=64 dropout=0.05 LoRA hyperparameters as #109.
      - Upload trained marker LoRA to HF Hub under `cp_marker_<src>_ep<step>_s42` for future reuse.
    - The 7 ep0 marker LoRAs already on HF Hub (`cp_marker_<src>_ep0_s42`) are reused; only the 70 (epoch≠0) variants are trained.
    - Parallelizable across 8 GPUs (one state per GPU subprocess at a time).
    
    ### Phase 0.5: leakage measurement at 11 targets (~6 GPU-h)
    - For each of 71 states (= 70 above + 7 ep0 already-existing):
      - Load base + convergence adapter + marker LoRA, merge.
      - Generate 11 personas × 20 questions × 10 completions @ temp=1.0, max_tokens=512, seed=42 (matches #109).
      - Compute marker rate = % completions containing `[ZLT]` substring (case-insensitive) per (source, target) cell.
      - Write `eval_results/causal_proximity/strong_convergence/<source>/checkpoint-<step>/marker_eval.json` per state.
    - Upload the entire `causal_proximity/strong_convergence/` directory as a WandB Artifact named `causal_proximity_strong_convergence_v1` so future analyses don't hit the same gap (addresses experimenter's `feedback_cached_data_assumptions.md`).
    
    ### Phase 1: JS sweep (unchanged from plan v3, ~2.5 GPU-h)
    - Existing `scripts/run_issue228_sweep.py` + `compute_js_convergence_228.py` + `aggregate_issue228.py` + `plot_issue228_figures.py` execute as designed.
    
    ### Compute
    - Pod: 8× H100 (provisioned, IP 103.207.149.77:12805).
    - Wall time: ~3-4 h on 8× H100 (Phase 0 ~2h, Phase 0.5 ~45min, Phase 1 ~25min, aggregate+plot ~5min).
    - GPU-hours budgeted: ~25.
    - `compute:medium` (5-20 GPU-h boundary; explicit user override per CLAUDE.md "Cost gates").
    
    ### Implementation deltas vs v3
    New scripts (to be written by implementer round 3):
    - `scripts/train_marker_loras_228.py` — per-state marker LoRA training (parameterized over 70 (source, ckpt) pairs; idempotent on existing HF artifacts; uploads on completion).
    - `scripts/measure_leakage_228.py` — per-state leakage measurement at 11 targets (writes to the cache path the existing aggregator expects).
    - Extend `run_issue228_sweep.py` to orchestrate Phase 0 → Phase 0.5 → Phase 1 with N-way GPU sharding.
    - Persist `causal_proximity_strong_convergence_v1` WandB Artifact at end of Phase 0.5.
    
    Existing scripts unchanged: `compute_js_convergence_228.py`, `aggregate_issue228.py`, `plot_issue228_figures.py`, `tests/test_issue228_aggregator.py`.
    
    ### Standing caveats (carried into clean-result)
    v3 caveat #1 (cross-state correlation between JS state and leakage state) **goes away** — both are now measured on the same convergence + marker-LoRA model state. Causal claims can be strengthened. The other v3 caveats (single seed, [ZLT] FP, villain dropout, prompt-distance interaction, effective N at n=71, power note) carry forward.
    
    ### Approval
    
    User has given explicit override. Advancing to `status:implementing` for round 3 (marker-LoRA + leakage scripts). Round 3 is the LAST allowed implementer round before `status:blocked`. Existing branch `issue-228` stays as the work tree; new scripts add to the same PR #242.
    <!-- /epm:plan -->

15. epm:experiment-implementation2026-05-05T04:46:09.000Z· system
    <!-- epm:experiment-implementation v3 --> ## Implementation Report — round 3 **Status:** READY-FOR-REVIEW ### Files ad…

    <!-- epm:experiment-implementation v3 -->
    ## Implementation Report — round 3
    
    **Status:** READY-FOR-REVIEW
    
    ### Files added / modified
    
    | File | Lines | What |
    |---|---|---|
    | `scripts/train_marker_loras_228.py` | +516 | NEW. Phase 0 single-state marker LoRA training. Idempotent on HF Hub. |
    | `scripts/measure_leakage_228.py` | +452 | NEW. Phase 0.5 single-state leakage at 11 targets. Aggregator-compatible JSON. |
    | `scripts/run_issue228_sweep.py` | +402 / -168 | EXTENDED. `--phase 0|0.5|1|all` routing + WandB artifact upload after Phase 0.5. |
    | `scripts/aggregate_issue228.py` | +18 / -8 | FIXED. `_c3_villain_out` keys `n70/n60` → `n71/n61` (round-2 NIT carryover). |
    | `tests/test_issue228_phase0_phase05.py` | +373 | NEW. 9 tests: schema round-trip + HF idempotency + phase routing + dry-run. |
    | `tests/test_issue228_aggregator.py` | +9 / -2 | UPDATED. Renamed `n70/n60` → `n71/n61` in C3 test to match aggregator. |
    
    Total: 1780 insertions, 168 deletions across 6 files.
    
    ### Reuse map: where #109/#112 logic was vendored
    
    Hyperparameters and protocol are reused, not re-invented:
    
    * **Marker LoRA hyperparameters** — sourced from `scripts/run_single_token_multi_source.py:55-66` (`BEST_LR=5e-6`, `BEST_EPOCHS=20`) and verified against the existing `cp_marker_<src>_ep0_s42/adapter_config.json` on HF Hub (`r=32`, `lora_alpha=64`, `lora_dropout=0.05`, `use_rslora=true`, all 7 target modules — base_model=`Qwen/Qwen2.5-7B-Instruct`).
    * **Marker training data** — reuses `run_leakage_v3_onpolicy.generate_deconfounded_marker_data`'s 200-positive + 200×2-negative shape. Generalised to non-V3 source personas via `_source_prompt_for(source)` falling through to `compute_js_convergence_228.EXTRA_SOURCE_PROMPTS`. Re-uses `generate_and_cache_onpolicy_data` for cached on-policy completions and falls through to `generate_onpolicy_completions` for the nurse row.
    * **`marker_only_loss=True, marker_text="[ZLT]", marker_tail_tokens=0`** — vendored from `run_single_token_sweep.py:163-166` and `run_single_token_multi_source.py:265-267`.
    * **Leakage protocol** — bit-identical to `scripts/eval_causal_ckpt.py:108-153` (11 personas × 20 questions × 10 completions, temp=1.0, top_p=1.0, max_tokens=512, seed=42; case-insensitive `[ZLT]` substring match).
    * **Convergence-adapter merge path** — same shape as `compute_js_convergence_228._download_and_merge_adapter:217-283` (snapshot_download → PeftModel → merge_and_unload → save_pretrained → reload).
    * **`use_rslora=True`** — automatic via `train_lora`'s default in `src/explore_persona_space/train/sft.py:298`. **Critical:** without rsLoRA the merged adapter would be ~5.7× too weak (BLOCKER-1 in `rerun_arms_ac.py:204-208`).
    
    ### Plan adherence (round-3 scope)
    
    * **Script 1 — `train_marker_loras_228.py`** — DONE.
      - HF-Hub idempotency via `HfApi.list_repo_files` + `target_file in files` check (skips silently on listing failure rather than blocking) ✓
      - Convergence-adapter merge into `/workspace/tmp/issue228_markerlora/<src>_ckpt<step>_merged/` ✓
      - Marker LoRA training with #109 hyperparameters, marker_only_loss=True ✓
      - HF upload via `train_lora`'s built-in `hf_upload=True, hf_path_in_repo=adapters/cp_marker_<src>_ep<step>_s<seed>` ✓
      - Cleanup of merged dir + adapter dir at exit ✓
    
    * **Script 2 — `measure_leakage_228.py`** — DONE.
      - Idempotency on `marker_eval.json` ✓
      - For `step > 0`: convergence + marker adapter merge into base; for `step == 0`: marker-only on base ✓
      - vLLM batched gen 11×20×10 @ temp=1.0, seed=42 — matches `eval_causal_ckpt.py` ✓
      - Schema `{persona: {"rate", "hits", "total"}, "_meta": {...}}` — round-tripped through aggregator's `_leakage_rate` in `test_marker_rates_schema_roundtrips_through_aggregator` ✓
      - `raw_completions.json` persisted alongside for C4 (temp=1) reanalysis ✓
      - Cleanup of merged dir at exit ✓
    
    * **Script 3 — extended `run_issue228_sweep.py`** — DONE.
      - `--phase 0|0.5|1|all` ✓ (default = `all`)
      - Per-phase enumeration / is_done / build_cmd via `PHASE_REGI

16. epm:fact-check-v2026-05-05T04:48:15.000Z· system
    <!-- epm:fact-check-v4 v1 --> ## Fact-check on plan v4 (parallel review — for code-reviewer round 3 to consider) A sepa…

    <!-- epm:fact-check-v4 v1 -->
    ## Fact-check on plan v4 (parallel review — for code-reviewer round 3 to consider)
    
    A separate fact-check pass on the v4 plan draft found the following factual corrections. Some have already been resolved by the in-flight implementer-3 / code-review-3 session; others remain operational concerns the reviewer should verify.
    
    ### Verified ✓ (no action)
    - Commit `c708b62` exists; `run_causal_proximity.py` at 1544 lines with `build_marker_data`, `train_marker_lora`, `run_arm_b` at the cited locations (off-by-≤8).
    - Marker-LoRA hyperparameters bit-identical: `lr=5e-6`, `epochs=20`, `marker_only_loss=True`, `marker_text="[ZLT]"`, `marker_tail_tokens=0`, LoRA `r=32 α=64 dropout=0.05`, `batch=4×4=16`, `max_length=1024`, `warmup_ratio=0.05`, `seed=42`.
    - All 7 source convergence adapters have HF Hub checkpoints at steps [200, 400, ..., 2000].
    - #109 headline cells (villain peak 73.2%, med doc 59.5%, comedian 42.5%, KT 32.5%) all match the issue body within ±0.5pp.
    
    ### Findings code-reviewer should verify (factual gaps in plan v4 prose)
    
    **[BLOCKER if not handled]** `build_marker_data` produces **~600 examples per state** (200 positive + 2 negative personas × 200 each = 400 negative), NOT the 400 quoted in the plan §5.2 / §10. Any pre-flight `assert 380 ≤ N ≤ 420` will FAIL on every state. Wall-time math is internally consistent on optimizer steps (still ~750), but the prose count is wrong. **Verify the implemented `train_marker_loras_228.py` doesn't enforce a 380-420 range.**
    
    **[OPERATIONAL]** On-policy completion cache `data/leakage_v3_onpolicy/onpolicy_cache/completions_<persona>.json` is **NOT on the HF Hub data repo** (`superkaiba1/explore-persona-space-data`, 0/628 files match). `pod.py sync data --pull` will not fetch it. Inline regeneration (~5 min/source × 7 sources ≈ 35 min upfront, not budgeted in §11) is mandatory. The implementer's worker has the inline-regenerate fallback already (calls `generate_and_cache_onpolicy_data`), so the experiment is still executable.
    
    **[OPERATIONAL]** Per-state wall-time of 14-16 min is ~30-50% optimistic. Real per-state likely **18-25 min** (750 optimizer steps × grad_accum=4 = 3000 forward passes × 0.4-0.7 s/forward on H100 7B+LoRA bf16 with grad-checkpointing). 70 states × 22 min ≈ **~26 GPU-h**, not 17.5. On 8× H100: **~3-4 h wall**, not 2.5-3 h. Still well within `inf-70b` 7-day TTL — flag for monitoring expectations only.
    
    **[UNVERIFIED]** Standing caveat #10 / sanity 6.1#7 reproducibility tolerance of ±15pp vs #109 published is engineering judgment, not empirically grounded. vLLM 0.8.0 (#109) → 0.11+ (current) drift is unmeasured.
    
    ### Already resolved ✓
    - File is `train_marker_loras_228.py` (plural), not the plan's singular naming — implementer-3 used the existing file.
    - Pod is 8× H100, not 1× (per task #17).
    - Sweep coordinator `run_issue228_sweep.py` is 628 lines (plan said 366) — already updated.
    
    Note: this fact-check was generated against `.claude/plans/issue-228-v4-draft.md` from a separate adversarial-planner pass that ran in parallel with implementer-3. The `epm:plan v2` already posted is the canonical pre-registration; flagging this only so code-reviewer round 3 can apply these checks against the implemented code.
    <!-- /epm:fact-check-v4 -->

17. epm:code-review2026-05-05T04:54:12.000Z· system
    <!-- epm:code-review v3 --> ## Code-Reviewer Verdict — CONCERNS (round 3) ### Verdict: CONCERNS — proceed to launch No…

    <!-- epm:code-review v3 -->
    ## Code-Reviewer Verdict — CONCERNS (round 3)
    
    ### Verdict: CONCERNS — proceed to launch
    
    No BLOCKERs found. R3 scope (Phase 0 marker LoRA training + Phase 0.5 leakage measurement + sweep extension to 0/0.5/1 + n70/n60→n71/n61 NIT fix) implements the plan v4 addendum correctly. Hyperparameters bit-match #109/#112. Schema round-trip through aggregator verified. Lint clean, 28/28 tests pass on my run, dry-run produces correct state counts (70/77/71). One MAJOR concern (cross-worker race on shared marker dataset path) and a few MINORs that do NOT block launch.
    
    ### Per-section verdicts
    
    | Section | Verdict | Notes |
    |---|---|---|
    | **A. `train_marker_loras_228.py`** | PASS-with-major | A1-A7 verified; A4 hyperparameters bit-match #109; **A5 race-condition concern (see Major-1)** |
    | **B. `measure_leakage_228.py`** | PASS | B1-B7 all verified; schema round-trip test exercises the real loader, not a mock |
    | **C. Sweep extension** | PASS | C1-C5 verified; dry-run prints 70/77/71 (consistent with implementer's report) |
    | **D. NIT fix + tests** | PASS | n70→n71 / n60→n61 rename consistent across aggregator + test fixture; 28/28 tests pass |
    | **E. Style/integration** | PASS | Lint + format clean on all 6 files; round-2 JS scripts untouched; no try/except: pass |
    
    ### Findings
    
    #### Major (worth flagging but does not block launch)
    
    * **Major-1 — `train_marker_loras_228.py:261`: cross-worker race on shared dataset file.**
      - `out_path = V3_DATA_DIR / f"marker_deconfounded_{source}_s{seed}_for228.jsonl"` does NOT include `step` in the filename.
      - With 8 concurrent workers and 10 ckpt steps per source, two workers on the same source (e.g. `villain ckpt-200` and `villain ckpt-400`) will overwrite the same file mid-flight.
      - The contents may differ across workers because `select_negative_personas` (line 173) seeds via `random.Random(hash(source) + 42)`, and Python 3.11 randomizes string `hash()` per-process (no `PYTHONHASHSEED=0` is set in the subprocess env or in `_bootstrap.py`).
      - Practical impact is small: the racing window is narrow (write → immediate `load_dataset(...)` read), positive examples are stable, and `train_lora` re-reads the file once at startup. But the Phase 0 invariant "same training dataset for all checkpoints of a given source" is technically violated.
      - **Suggested fix (do this BEFORE launch only if cheap):** add `step` to the filename: `f"marker_deconfounded_{source}_ep{step}_s{seed}_for228.jsonl"`. Or set `PYTHONHASHSEED=42` in `_run_one_state`'s `env` dict (~3 LOC in `run_issue228_sweep.py:264-266`).
      - **Risk if shipped as-is:** the marker LoRAs may differ slightly between checkpoints of the same source due to negative-persona randomness. Downstream H1 leakage signal should still hold (negative personas only affect the "no-marker" half of the data) but reproducibility is mildly degraded.
    
    #### Minor (does NOT block launch)
    
    * **Minor-1 — `train_marker_loras_228.py:402-405`: `gpu_id` override is silent under `CUDA_VISIBLE_DEVICES=0`.** When the parent narrows `CUDA_VISIBLE_DEVICES=N` for one GPU, the worker overwrites `gpu_id=0`. This is correct behavior but the `if len(visible) == 1:` check is fragile — if the parent passes `CUDA_VISIBLE_DEVICES="0,1"` for any reason, the first GPU is silently used. Not exercised by the current sweep but worth a comment.
    
    * **Minor-2 — `train_marker_loras_228.py:411-416`: merged-dir reuse without freshness check.** When `merged_dir` already exists with a `config.json`, the worker reuses it. Idempotency is good for crash-recovery, but if the convergence adapter on Hub were updated mid-sweep, the local stale merge would silently propagate. Low-risk for this experiment (all adapters are immutable per #112's contract) but worth a `--force-remerge` flag for future sweeps.
    
    * **Minor-3 — `measure_leakage_228.py:404`: `raw_completions.json` is written without `indent=` so it's one giant line.** Fine for machine consumption, but inconsistent with `m

18. epm:launch2026-05-05T04:55:18.000Z· system
    <!-- epm:launch v2 --> ## Launch — round 2 (full path-A pipeline) - **Branch:** `issue-228` @ `99a3716` (post-round-3 i…

    <!-- epm:launch v2 -->
    ## Launch — round 2 (full path-A pipeline)
    
    - **Branch:** `issue-228` @ `99a3716` (post-round-3 implementation)
    - **Code-review:** `epm:code-review v3` CONCERNS / proceed (https://github.com/superkaiba/explore-persona-space/issues/228#issuecomment-4376628962)
    - **Pod:** `epm-issue-228` (8× H100, 103.207.149.77:12805)
    - **Phases:**
      - Phase 0: train 70 marker LoRAs (~14 GPU-h)
      - Phase 0.5: measure leakage at 11 targets per state, write 71 marker_eval.json files, upload to WandB Artifact `causal_proximity_strong_convergence_v1` (~6 GPU-h)
      - Phase 1: existing JS sweep on convergence-only adapters (~2.5 GPU-h)
      - Aggregate + plot (~5 min)
    - **Wall estimate:** ~3-4 h on 8× H100 (~25 GPU-h total)
    - **Pre-launch hot-fix (≤10 LOC):** `PYTHONHASHSEED=42` injected into worker subprocess env to address reviewer's MAJOR (cross-worker dataset-shuffling race condition).
    - **WandB run URL:** TBD (experimenter posts on launch)
    - **Log path:** /workspace/issue228_sweep.log
    <!-- /epm:launch -->

19. epm:hot-fix2026-05-05T04:57:23.000Z· system
    <!-- epm:hot-fix v1 --> ## Hot-fix: PYTHONHASHSEED=42 in worker env **Commit:** `356b6b5` (issue-228 branch) **File:**…

    <!-- epm:hot-fix v1 -->
    
    ## Hot-fix: PYTHONHASHSEED=42 in worker env
    
    **Commit:** `356b6b5` (issue-228 branch)
    **File:** `scripts/run_issue228_sweep.py` (1 line added)
    **Justification:** Round-3 reviewer flagged a cross-worker race on `marker_deconfounded_<source>_s42_for228.jsonl` because Python's `hash()` is per-process randomized. Pinning `PYTHONHASHSEED=42` in the worker subprocess env makes hash() deterministic across all 8 GPU workers, closing the race. Hot-fix bar met: 1 line, no logic change, single `Edit` to `_run_one_state` env construction.
    
    ```diff
    diff --git a/scripts/run_issue228_sweep.py b/scripts/run_issue228_sweep.py
    index 003f81e..2b88f51 100644
    --- a/scripts/run_issue228_sweep.py
    +++ b/scripts/run_issue228_sweep.py
    @@ -263,6 +263,7 @@ def _run_one_state(
     ) -> int:
         env = os.environ.copy()
         env["CUDA_VISIBLE_DEVICES"] = str(gpu_id)
    +    env["PYTHONHASHSEED"] = "42"  # round-3 hot-fix: stabilize hash() across workers (#228)
         log_path.parent.mkdir(parents=True, exist_ok=True)
         logger.info("[gpu=%d] launching %s ckpt-%d -> %s", gpu_id, source, step, log_path)
         t0 = time.time()
    ```
    
    Pod `epm-issue-228` pulled to `356b6b5`. Proceeding with marker-mask sanity check + #142 baseline pull + launch.

20. epm:progress2026-05-05T05:06:41.000Z· system
    <!-- epm:progress v1 --> ## Sweep launched **Pod:** `epm-issue-228` (8× H100, 100% free pre-launch) **PID (coordinator…

    <!-- epm:progress v1 -->
    
    ## Sweep launched
    
    **Pod:** `epm-issue-228` (8× H100, 100% free pre-launch)
    **PID (coordinator):** 18714 → 18717 (uv run wrapper) → 8 worker subprocesses
    **Log:** `/workspace/issue228_sweep.log`
    **Worker logs:** `/workspace/explore-persona-space/eval_results/issue_228/_worker_logs/`
    **Launch time:** 2026-05-05 05:05:35 UTC
    **Branch HEAD:** `356b6b5` (incl. PYTHONHASHSEED hot-fix v1)
    
    ### Phases dispatched
    - Phase 0 (marker LoRA training): 70 states across 8 GPUs — IN PROGRESS
    - Phase 0.5 (leakage measurement): 77 states — pending
    - Phase 1 (JS sweep): 71 states — pending
    
    ### Pre-launch fix-ups (not code changes)
    1. **vLLM/transformers compat patch** — known issue from `feedback_transformers55_vllm_compat.md` memory: vLLM 0.11.0 + transformers 5.5.0 + tqdm 4.67.3 fail to load any tokenizer with `AttributeError: Qwen2Tokenizer has no attribute all_special_tokens_extended`. Applied the canonical `.pth` patch (`_epm_157_compat.py` + `_epm_157_compat.pth`) into pod's site-packages. Smoke-tested: `OK gen: ", I'm trying to create a program"` from greedy Qwen2.5-7B. **This is an environment patch, not a code change** — pods are ephemeral and the patch dies with the pod.
    2. **Disk cleanup** — prior failed sweep left 176 GB of stale merged dirs in `/workspace/tmp/issue228_markerlora/` from interrupted runs. Cleaned to recover space (200G→176G free pre-launch). Per-state cleanup logic exists in `train_marker_loras_228.py:469` (post-success), so steady-state disk should not regrow.
    
    ### ETA
    - Phase 0: ~2 h (70 states / 8 GPUs ≈ 9 batches × ~13 min/state)
    - Phase 0.5: ~45 min
    - Phase 1: ~25 min
    - Aggregate + plot: ~5 min
    - **Total: ~3.0–3.5 h** (target completion ≈ 08:30 UTC)
    
    ### Monitoring schedule
    First t+5min, then t+30min, t+1h, t+2h, t+2.75h, t+3h, t+3.5h. Will surface OOMs / errors immediately if any.
    
    ### Sanity checks PASSED before launch
    - Marker-mask real-tokenizer test: only `Ġ[ Z LT ]` constituent tokens masked, trailing `]` and newline correctly unmasked
    - #142 JS baseline (`eval_results/js_divergence/divergence_matrices.json`, 735 KB) staged on pod for the aggregator
    - Dry-run confirmed work plan: Phase 0 = 70, Phase 0.5 = 77, Phase 1 = 71 states

21. epm:failure2026-05-05T05:13:38.000Z· system
    <!-- epm:failure v3 --> ## Bounce back to status:implementing — round 4 **Run aborted. Multiple cascading infrastructu…

    <!-- epm:failure v3 -->
    
    ## Bounce back to status:implementing — round 4
    
    **Run aborted. Multiple cascading infrastructure issues exceed hot-fix bar.**
    
    ### What worked
    - PYTHONHASHSEED hot-fix v1 committed at `356b6b5`
    - vLLM/transformers compat `.pth` patch installed on pod (env-side, not code; per `feedback_transformers55_vllm_compat` memory)
    - Disk cleanup of stale 176 GB merged dirs from prior failed runs
    - Marker-mask real-tokenizer sanity check PASSED
    - #142 JS baseline staged on pod
    
    ### Failure mode
    After launch, **5 of 8 first-batch workers failed at vLLM init** with:
    ```
    ValueError: Free memory on device (31.43/79.18 GiB) on startup is less than
    desired GPU memory utilization (0.6, 47.51 GiB).
    ```
    
    3 workers (ckpt-200, 400, 800) succeeded into vLLM gen; 5 (600, 1000, 1200, 1400, 1600) failed.
    
    ### Root causes (interacting)
    
    **1. PEFT merge leaves CUDA cache that vLLM can't fit alongside.** `_merge_convergence_adapter` in `scripts/train_marker_loras_228.py:320` calls `AutoModelForCausalLM.from_pretrained → PeftModel.from_pretrained → merge_and_unload → save_pretrained`, then `del peft_model, merged, base, tokenizer` + `gc.collect()` + `torch.cuda.empty_cache()` (lines 365-367). Despite the cleanup, the parent's allocator pool retains ~14-28 GiB cached. When vLLM `LLM(...)` then spawns its EngineCore subprocess on the same GPU, the subprocess sees only 31-51 GiB free; `gpu_memory_utilization=0.60` (from `run_leakage_v3_onpolicy.py:253`, `os.environ.get("VLLM_GPU_MEM_UTIL", "0.60")`) wants 47.5 GiB → fails on the unlucky workers.
    
    This is a **fundamental resource accounting issue** in the sweep design. It's not a 1-line tweak: lowering VLLM_GPU_MEM_UTIL via env (launch-time override, no code change) reduces vLLM's footprint but also reduces KV cache headroom — and after vLLM teardown, training (HF Trainer LoRA) still needs another ~25-30 GiB. The fix needs design-level coordination, not a magic-number-tweak.
    
    **2. GPU 0 has zombie CUDA allocations consuming 79 GiB.** From an earlier vLLM smoke test on GPU 0, two zombie PIDs (`2322382`, `2322743`) survived process death and now permanently hold 47982 + 31014 MiB on physical GPU 0. Per my agent memory: "Zombie CUDA allocations survive process death. Only fix is container restart." `nvidia-smi --gpu-reset` returns `Not Supported` in this container.
    
    This means GPU 0 is unusable until the pod is restarted, leaving us with 7 of 8 GPUs.
    
    ### Why this is bounce-back, not hot-fix
    
    Three interacting problems:
    
    1. **Resource accounting** in the sweep — design-level fix needed (e.g., release CUDA pool between merge and vLLM via subprocess isolation, OR move merge into a separate child process, OR drop `gpu_memory_utilization` to 0.35 AND verify training fits in remaining 51 GiB AND verify on-policy gen completes in the smaller KV cache without thrashing). Each of these is more than 10 LOC and changes logic.
    
    2. **Lost GPU** — needs pod stop/resume to clear zombie. `/issue` skill territory, not mine.
    
    3. **Concurrency safety on `onpolicy_cache/completions_<source>.json`** — TOCTOU race between 10 workers for the same source: all check `cache_path.exists()` simultaneously, all spin up their own vLLM gens, only the first `json.dump` wins; the others waste 5-10 min of compute and risk corrupting the cache file mid-write. Round-3 review didn't flag this because the failure mode looks identical to OOM. The fix is per-source filelock or a Phase-0a step that pre-generates all caches BEFORE Phase 0 launches its 70-state fan-out.
    
    ### Proposed fixes for round 4 (`experiment-implementer`)
    
    **Required (blockers):**
    
    A. **Phase 0a: serial cache pre-generation.** Before Phase 0 launches its 70-way fan-out, generate the 7 `onpolicy_cache/completions_<source>.json` files serially (1 GPU at a time). Eliminates the TOCTOU race AND lets Phase 0 workers hit the cache hot path (no vLLM in worker → no merge-vs-vLLM memory contention).
    
    B. **Drop in-worker vLLM gen.** With Phase 0a populating the cache

22. epm:experiment-implementation2026-05-05T05:29:44.000Z· system
    <!-- epm:experiment-implementation v4 --> ## Implementation Report — round 4 **Status:** READY-FOR-REVIEW User-approve…

    <!-- epm:experiment-implementation v4 -->
    ## Implementation Report — round 4
    
    **Status:** READY-FOR-REVIEW
    
    User-approved exception to the 3-round cap: this is structural debugging of round-3's launch failure, not iteration on a failing implementation.
    
    ### Files added / modified vs round 3
    
    | File | Lines (vs R3) | What |
    |---|---|---|
    | `scripts/pregenerate_onpolicy_cache_228.py` | NEW (+178) | Phase 0a worker. One source per invocation, one vLLM session, no PEFT merge. |
    | `scripts/run_issue228_sweep.py` | +95 / -10 | Adds Phase 0a to `PHASE_REGISTRY` with `force_serial=True`; updates `--phase` choices to `{0a, 0, 0.5, 1, all}`; `--phase all` dispatches `[0a, 0, 0.5, 1]` in order. |
    | `scripts/run_leakage_v3_onpolicy.py` | +24 / -7 | `generate_and_cache_onpolicy_data` now wraps the existence-check + write in a per-source `filelock.FileLock`; cache write goes via tmp file + `os.replace` for atomicity. |
    | `scripts/train_marker_loras_228.py` | +47 / -40 | `_ensure_completions_cache` is now READ-ONLY. Raises `FileNotFoundError` (with explicit Phase-0a invocation hint) when the cache is missing. Removes silent in-worker vLLM regeneration. |
    | `tests/test_issue228_phase0_phase05.py` | +274 / -8 | 10 new tests (cache fail-loud, nurse dual-cache, filelock serialisation, atomic rename, Phase 0a registry shape + force_serial, four-phase order, state-count regression). Updated 4 existing tests for Phase 0a. |
    
    Total round-4 delta: **750 insertions, 94 deletions across 5 files.** `git diff --stat HEAD~1`:
    
    ```
     scripts/pregenerate_onpolicy_cache_228.py | 188 ++++++++++++++++++++++
     scripts/run_issue228_sweep.py             | 109 +++++++++---
     scripts/run_leakage_v3_onpolicy.py        |  81 ++++-----
     scripts/train_marker_loras_228.py         |  87 ++++------
     tests/test_issue228_phase0_phase05.py     | 364 ++++++++++++++++++++++++++++++++++++++++-
     5 files changed, 750 insertions(+), 94 deletions(-)
    ```
    
    ### Diagnosis (per the experimenter's R3 failure marker)
    
    The round-3 failure had a single root cause that interacted with two other issues:
    
    **Root cause (Fix A target):** in `train_marker_loras_228.py:419` (R3 line numbering), when the on-policy cache for a source was missing, `_ensure_completions_cache` called `generate_and_cache_onpolicy_data` (lines 273-314 R3), which internally spawns vLLM via `generate_onpolicy_completions` → `LLM(...)`. By that point in the worker's lifecycle, `_merge_convergence_adapter` (line 320 R3) had already run `AutoModelForCausalLM.from_pretrained → PeftModel.from_pretrained → merge_and_unload`, leaving 14-28 GiB committed in the parent's CUDA allocator pool. `del peft_model, merged, base, tokenizer; gc.collect(); torch.cuda.empty_cache()` (lines 365-367 R3) does NOT release the allocator pool back to the device. vLLM's EngineCore subprocess then sees physical-device free memory of only 31-51 GiB; with `gpu_memory_utilization=0.60` (47.5 GiB target, set in `run_leakage_v3_onpolicy.py:253` via `VLLM_GPU_MEM_UTIL` env), 5 of 8 workers failed to allocate.
    
    **Why R3 dry-run + R3 code-review didn't catch it:** the dry-run never executes `_ensure_completions_cache`; it only enumerates command lines. Code-review identified a related cross-worker race on `marker_deconfounded_*.jsonl` (Major-1 in `epm:code-review v3`) but didn't model the cache-miss path that actually crashed at runtime.
    
    **Where the same-process pattern lives elsewhere:**
    - Phase 0.5 (`measure_leakage_228._merge_two_adapters` lines 159-201 → `_generate_completions` lines 207-269): same merge-then-vLLM pattern in one process. NOT a round-4 fix because (a) it uses `gpu_memory_utilization=0.85` default (more headroom — vLLM only needs ~67 GiB instead of 47 GiB), (b) only ONE merge happens before vLLM (not chained merges), (c) the round-2 PASS verdict on the JS sweep code (`compute_js_convergence_228.py`) confirmed the same merge-then-vLLM pattern works at 8-way parallelism with `gpu_memory_utilization=0.85`. If Phase 0.5 OOMs at launch, the exper

23. epm:code-review2026-05-05T05:39:26.000Z· system
    <!-- epm:code-review v4 --> ## Code-Reviewer Verdict — PASS **Diff size:** +750 / -94 across 5 files (1 new, 4 modified…

    <!-- epm:code-review v4 -->
    ## Code-Reviewer Verdict — PASS
    
    **Diff size:** +750 / -94 across 5 files (1 new, 4 modified)
    **Plan adherence:** COMPLETE (Fix A + B + C all present and correctly wired)
    **Tests:** 38/38 #228 tests pass
    **Lint:** PASS (`ruff check` + `ruff format --check` clean on all 5 files)
    **Security sweep:** CLEAN (no hardcoded secrets, no shell injection, no silent exception swallows)
    
    ---
    
    ### E. Load-bearing question — on-policy cache is BASE-model generated. ASSUMPTION CONFIRMED
    
    The implementer's design hinges on: "the on-policy completion cache is generated by the BASE Qwen-2.5-7B-Instruct model, not by any convergence-merged variant." I verified this rigorously:
    
    - **`scripts/run_leakage_v3_onpolicy.py:282-290`** — `LLM(model=BASE_MODEL, ...)` where `BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"` is hard-coded at line 54. `generate_onpolicy_completions` accepts NO `model_path` argument; it always loads the bare base model.
    - **`scripts/run_leakage_v3_onpolicy.py:367-373`** — `generate_and_cache_onpolicy_data` calls `generate_onpolicy_completions` with no model override.
    - **`scripts/pregenerate_onpolicy_cache_228.py:130, 138, 151`** — Phase 0a delegates to those exact functions; never accepts or threads through a model path.
    
    **Consumer pattern verified (E2):**
    - `train_marker_loras_228._build_marker_training_data` at `scripts/train_marker_loras_228.py:222-237`: positive examples are `_make_example(source_prompt, question, f"{comp}\n\n{MARKER_TOKEN}")` — the cached completion is a TEXT PREFIX with `\n\n[ZLT]` appended. With `marker_only_loss=True` (line 474), training loss is masked to only `[ZLT]` tokens; gradients never flow through `comp`. So the cache's role is "what does the assistant turn LOOK LIKE before the marker fires" — pattern (a) in the brief.
    - **Parity with #112 confirmed**: `scripts/run_single_token_multi_source.py:93-94, 232-233` (DATA_DIR shared with v3, line 47) reuses the same `generate_and_cache_onpolicy_data` -> `generate_deconfounded_marker_data` chain. #112 ALSO trained marker LoRAs on top of the convergence-merged model with a base-model-generated cache. R4 reproduces #112's design exactly.
    
    **Verdict: implementer's flagged assumption is correct.** R4 is structurally sound.
    
    ---
    
    ### A. `scripts/pregenerate_onpolicy_cache_228.py` — PASS
    - A1: No PEFT merge in this script. Imports only `vllm` (transitively via `generate_*` calls); no `peft`/`PeftModel`/`merge_and_unload`. Verified via grep: zero matches in this file.
    - A2: Schema match. V3 path (line 130) writes `completions_<src>.json` via the canonical `generate_and_cache_onpolicy_data` (run_leakage_v3_onpolicy.py:340). Nurse path (line 161-164) writes the same shape (`{persona: {question: [completions]}}`) to `completions_nurse.json`. Consumer at `train_marker_loras_228.py:298-329` reads exactly this schema.
    - A3: Atomic write. V3 path inherits the canonical atomic write (`run_leakage_v3_onpolicy.py:377-380`). Nurse path uses `tmp_path = cache_path.with_suffix(".json.tmp")` -> `os.replace` (lines 161-164).
    - A4: Idempotency. `_is_cache_valid` (line 84-94) checks file exists, non-empty, parseable JSON, dict-typed, len>0. Stricter than the coordinator's `_phase0a_state_done` (good defense-in-depth).
    
    **Note (non-blocking):** When Phase 0a runs `nurse` BEFORE `villain` in alphabetical order (which it does — sort gives `comedian, kindergarten_teacher, librarian, medical_doctor, nurse, software_engineer, villain`), the nurse worker generates `completions_villain.json` as a side effect (line 138). This is correct, and the coordinator's `_phase0a_state_done` for villain then short-circuits at dispatch time — confirmed safe under `force_serial=True`.
    
    ### B. `scripts/run_issue228_sweep.py` Phase 0a routing — PASS
    - B1: `--phase 0a`/`0`/`0.5`/`1`/`all` all parse via `argparse choices` (line 597-605). `all` runs `[0a, 0, 0.5, 1]` in order (verified by `test_phase_all_runs_four_phases_in_order`).
    - B2: `force_serial=True` for Phase 0a (line 27

24. epm:progress2026-05-05T05:47:19.000Z· system
    <!-- epm:progress v2 --> ## Launch (R4 second attempt) - **PID:** 2543 - **Pod:** epm-issue-228 (8× H100 at 103.207.14…

    <!-- epm:progress v2 -->
    
    ## Launch (R4 second attempt)
    
    - **PID:** 2543
    - **Pod:** epm-issue-228 (8× H100 at 103.207.149.77:13573)
    - **Branch / commit:** `issue-228` @ `0982b9b` (R4)
    - **Launch command:**
      ```
      .venv/bin/python scripts/run_issue228_sweep.py \
        --num-gpus 8 --phase all --seed 42 \
        --output-dir eval_results/issue_228 \
        --leakage-output-dir eval_results/causal_proximity/strong_convergence
      ```
    - **Log:** `/workspace/issue228_sweep.log` on the pod
    - **Launch:** 2026-05-05 05:44 UTC
    - **ETA:** ~3.7 h (≈ 09:30 UTC)
    
    ## 4-phase ordering
    
    1. **Phase 0a** (on-policy cache pre-gen) — **force-serial on 1 GPU** by design (R4 fix to address R3 OOM from in-process PEFT-merge + vLLM contention; subprocess-per-source so OS reclaims CUDA allocator). 7 sources × ~3 min ≈ 21 min.
    2. **Phase 0** (marker LoRA training, 70 states) — 8-way parallel ≈ 110 min.
    3. **Phase 0.5** (leakage measurement, 77 states) — 8-way parallel ≈ 60 min. Uploads `causal_proximity_strong_convergence_v1` Artifact when done.
    4. **Phase 1** (JS sweep, 71 states) — 8-way parallel ≈ 45 min.
    
    Aggregator + plotter ≈ 5 min after Phase 1. Aggregator hard-aborts if sanity 6.1#4 (MRE < 5%, ρ > 0.99 vs #142 baseline at `eval_results/js_divergence/divergence_matrices.json`) fails.
    
    ## Notes vs brief
    
    - The sweep coordinator does not take a `--js-baseline-142` flag; that comparison is wired in the aggregator script which reads the baseline JSON directly. Confirmed `divergence_matrices.json` (718 KB) survived stop/resume.
    - The brief gave `--leakage-dir`; correct flag is `--leakage-output-dir`. Corrected on launch.
    - `uv` is not on the resumed pod's PATH; using `.venv/bin/python` directly. The sweep coordinator uses `sys.executable` for its subprocess workers, so they inherit the venv python automatically.
    - Marker tokenization sanity: `[ZLT]` → ids `[85113, 27404, 60]` → `['[Z', 'LT', ']']`. Clean constituent tokens.
    - PYTHONHASHSEED=42 hot-fix from `356b6b5` is baked into `_run_worker` (line 335 of `run_issue228_sweep.py`).
    
    ## R3 → R4 architecture fix
    R3 OOM came from in-process PEFT merge + vLLM child not seeing the freed-but-fragmented allocator. R4 separates them via subprocess-per-source in Phase 0a, so each source's vLLM child starts with a fresh CUDA context.
    
    ## WandB
    URL will be added on first marker LoRA training step (Phase 0).
    

25. epm:failure2026-05-05T06:01:40.000Z· system
    <!-- epm:failure v4 --> ## Stop reason — second R4 launch attempt deadlocks at FileLock The R4 commit `0982b9b` introd…

    <!-- epm:failure v4 -->
    
    ## Stop reason — second R4 launch attempt deadlocks at FileLock
    
    The R4 commit `0982b9b` introduced **two redundant `filelock.FileLock` calls
    on the same path from different instances**. They deadlock at OS-level `flock(2)`
    for the inner-lock's `timeout=600s`, then raise `Timeout`. Worker never reaches
    vLLM init — GPU stays at 0 MiB the entire time.
    
    This bug is in the round-4 code that was supposed to fix R3's OOM, not in
    infra. Code-review R4 missed it because the two locks live in different files
    and the reviewer didn't trace the cross-file call chain.
    
    ## Reproduction (deterministic)
    
    ```bash
    # Direct reproduction without the sweep coordinator
    ssh epm-issue-228 'cd /workspace/explore-persona-space && \
      CUDA_VISIBLE_DEVICES=0 PYTHONHASHSEED=42 \
      .venv/bin/python -u scripts/pregenerate_onpolicy_cache_228.py \
      --source comedian --gpu-id 0'
    ```
    
    Output sequence:
    ```
    INFO 05-05 05:56:35 [__init__.py:216] Automatically detected platform cuda.
    2026-05-05 05:56:39 [INFO] [comedian] generating canonical 10-persona on-policy cache
    <HANG ~10 min, 0 MiB GPU, 13% CPU>
    <eventually filelock.Timeout from the inner FileLock>
    ```
    
    ## Root cause
    
    Two FileLocks on the **same path** but **different instances**:
    
    **Outer** — `scripts/pregenerate_onpolicy_cache_228.py`, lines 113-117 (added in R4):
    ```python
    lock_path = CACHE_DIR / f".{source}.lock"
    lock = FileLock(str(lock_path), timeout=60)
    with lock:
        ...
        generate_and_cache_onpolicy_data(source, gpu_id=gpu_id)  # <- enters inner
    ```
    
    **Inner** — `scripts/run_leakage_v3_onpolicy.py`, line 343 (added in R4 per the
    "round-4 concurrency guard" docstring):
    ```python
    with FileLock(str(lock_path), timeout=600):
        if cache_path.exists() and cache_path.stat().st_size > 0:
            ...
            return json.load(f)
        completions = generate_onpolicy_completions(...)
        ...
    ```
    
    `filelock.FileLock` IS re-entrant when the **same instance** is acquired
    twice (uses an internal `_lock_counter`). It is NOT re-entrant across
    **different instances** on the same path — the OS's `flock(2)` rejects the
    second exclusive lock. Verified with a 5-line direct test on the pod:
    
    ```python
    from filelock import FileLock
    with FileLock("/tmp/test.lock", timeout=2):
        with FileLock("/tmp/test.lock", timeout=2):  # raises Timeout after 2s
            ...
    ```
    
    ## Verification that the rest of the stack works
    
    To rule out vLLM / pod / driver issues, I ran a clean smoke test in a fresh
    subprocess that bypasses `pregenerate_onpolicy_cache_228.py` entirely:
    
    ```python
    # /tmp/test_vllm_init.py
    from vllm import LLM, SamplingParams
    llm = LLM(model="Qwen/Qwen2.5-7B-Instruct", dtype="bfloat16",
              gpu_memory_utilization=0.5, max_model_len=2048,
              max_num_seqs=64, seed=42)
    out = llm.generate(["Hello, world!"], SamplingParams(n=1, temperature=0.7, max_tokens=32))
    ```
    
    Result: vLLM imports in 1s, full LLM init in 50s, generation works
    ("Welcome to the Math Olympiad Prep course for high..."). vLLM v0.11.0 +
    torch.compile + CUDA graph capture all functional. **The bug is purely the
    filelock deadlock added in R4.**
    
    ## Proposed fix (for experiment-implementer)
    
    Pick **one** of these — both are surgical:
    
    ### Option A (preferred) — drop the inner lock
    
    In `scripts/run_leakage_v3_onpolicy.py::generate_and_cache_onpolicy_data`,
    remove the `with FileLock(str(lock_path), timeout=600):` block (the function
    runs unconditionally now that the outer caller serializes). Update the
    docstring comment "Concurrency guard (added round 4 of #228)" to point at the
    outer lock instead.
    
    This relies on the invariant that `generate_and_cache_onpolicy_data` is
    ONLY called from contexts that already hold the per-source filelock (today:
    `pregenerate_one_source` in `pregenerate_onpolicy_cache_228.py`). Verify no
    other callers — `grep -rn "generate_and_cache_onpolicy_data" scripts/ src/`
    must show no other uncovered call sites.
    
    ### Option B — pass the lock through
    
    Change `pregenerate_one_source` to acquire the lock

26. epm:code-review2026-05-05T06:10:56.000Z· system
    <!-- epm:code-review v5 --> ## Code-Reviewer Verdict — PASS Round-5 hot-fix review of commit `37720da` only. Scope: dro…

    <!-- epm:code-review v5 -->
    ## Code-Reviewer Verdict — PASS
    
    Round-5 hot-fix review of commit `37720da` only. Scope: drop the inner `filelock.FileLock` from `generate_and_cache_onpolicy_data` that deadlocked against the outer lock in `pregenerate_one_source` during the R4 launch.
    
    **Diff size:** +37 / -73 across 2 files (`scripts/run_leakage_v3_onpolicy.py`, `tests/test_issue228_phase0_phase05.py`)
    **Plan adherence:** COMPLETE — surgical deletion only; no scope creep
    **Tests:** PASS (38/38)
    **Lint:** PASS (ruff check + format)
    **Security sweep:** CLEAN
    
    ### 9-check verification
    
    | # | Check | Result | Evidence |
    |---|---|---|---|
    | 1 | Inner FileLock fully removed | PASS | `grep -in filelock\|FileLock\|lock_path` on `run_leakage_v3_onpolicy.py @ 37720da` returns only docstring lines 330–332 (prose) and unrelated `merge_lock_path` at line 629 (different function). No `from filelock import`, no `with FileLock(...)`, no `lock_path` variable in `generate_and_cache_onpolicy_data`. |
    | 2 | Cache-hit path works | PASS | Lines 339–342: `if cache_path.exists() and cache_path.stat().st_size > 0:` → `json.load(f)` → return. Covered by new `test_cache_hit_skips_generator`. |
    | 3 | Atomic tmp+rename preserved | PASS | Lines 372–376 still write to `.json.tmp` then `os.replace(tmp_path, cache_path)`. Covered by `test_filelock_uses_atomic_rename` (kept intact). |
    | 4 | Docstring updated | PASS | Lines 327–336 explicitly call out (a) caller-owned concurrency, (b) Phase 0a serial coordinator + outer `filelock` in `pregenerate_onpolicy_cache_228.py` are the guard, (c) inner lock removed because `FileLock` is per-instance reentrant only (R5 hot-fix), (d) atomic tmp+rename still protects readers. |
    | 5 | Test updated coherently | PASS | Old `test_filelock_serialises_concurrent_cache_writes` deleted. New `test_cache_hit_skips_generator` is a real invariant test: pre-writes cache, monkeypatches generator to raise on entry, asserts (a) returned payload equals on-disk payload, (b) `invocation_count == 0`. Not a tautology. |
    | 6 | Outer lock still operative | PASS | `pregenerate_one_source` in `pregenerate_onpolicy_cache_228.py` @ 37720da lines 95–115 is unchanged from pre-fix state: `FileLock(str(lock_path), timeout=60)` on `CACHE_DIR / f".{source}.lock"`. Diff scope limited to two files — outer-lock script untouched. |
    | 7 | No silent fallbacks | PASS | Diff is a pure deletion of the wrapping `with FileLock(...)` block + re-indent of its body. No bare `except`, no hardcoded paths added, no force-suppress flags. |
    | 8 | Tests pass | PASS | `uv run pytest tests/test_issue228_phase0_phase05.py tests/test_issue228_aggregator.py -v` → **38 passed in 20.94s** (19 phase0/phase05 + 19 aggregator). |
    | 9 | Lint clean | PASS | `ruff check`: "All checks passed!". `ruff format --check`: "2 files already formatted". |
    
    ### Concurrency story after the fix
    
    With the inner lock gone, the entire concurrency story for the on-policy cache is:
    
    1. **Phase 0a coordinator-level serialization** (`force_serial=True` per R4) — workers don't dispatch concurrent calls on the same source.
    2. **Outer `FileLock` in `pregenerate_one_source`** (timeout=60) — defends against accidental concurrent CLI invocations.
    3. **Atomic tmp+rename inside `generate_and_cache_onpolicy_data`** — defends readers who bypass the lock from observing a partial write.
    
    This is sufficient. The R4 deadlock root cause (two `FileLock` instances on the same path inside the same process) is structurally impossible now because there is only one lock instance.
    
    ### Minor observation (non-blocking, no action required)
    
    The docstring's phrasing "`filelock.FileLock` is only per-instance reentrant" is slightly hand-wavy — strictly, `filelock.FileLock` *is* reentrant for the same instance in the same thread; the deadlock arises because R4 had two *separate* `FileLock` instances on the same path. The user-facing rule the docstring is communicating ("don't take a second lock on this path inside the outer's critical se

27. epm:progress2026-05-05T06:26:54.000Z· system
    <!-- epm:progress v3 --> **Sweep launched (R5 — third launch attempt)** - **PID:** 11775 (parent uv run: 11774, python:…

    <!-- epm:progress v3 -->
    **Sweep launched (R5 — third launch attempt)**
    
    - **PID:** 11775 (parent uv run: 11774, python: 11778) on `epm-issue-228` (8× H100 at `103.207.149.77:13573`)
    - **Log:** `/workspace/issue228_sweep.log`
    - **Command:** `nohup uv run python scripts/run_issue228_sweep.py --num-gpus 8 --phase all --output-dir eval_results/issue_228 --leakage-output-dir eval_results/causal_proximity/strong_convergence --seed 42 > /workspace/issue228_sweep.log 2>&1 &`
    - **Branch:** `issue-228` @ `37720da` (R5 hot-fix removed inner FileLock that deadlocked R4); R4+R5 code-review verdicts: PASS
    - **Launch timestamp:** 2026-05-05 06:24:30 UTC
    
    **Phase 0a smoke test (`comedian` on GPU 0): PASSED**
    - `data/leakage_v3_onpolicy/onpolicy_cache/completions_comedian.json` — 14MB, 6000 completions written, exit 0 in 458s
    - **No deadlock.** R5 fix confirmed working.
    - Comedian cache is now persisted; sweep auto-skipped it (`comedian ckpt--1 already complete; skipping`).
    
    **Pre-flight artefacts (all survived stop/resume):**
    - `eval_results/js_divergence/divergence_matrices.json` — 718KB, present
    - `_epm_157_compat.{py,pth}` — present in venv
    - `data/leakage_v3_onpolicy/onpolicy_cache/` — only `completions_comedian.json` (no stale `.lock` files)
    
    **Hot-fixes applied in this session:**
    - `_epm_157_compat.py` (on-pod env patch only, not repo code): wrapped `apply()` in `contextlib.redirect_stdout(devnull)` so vLLM's import-time logging stops corrupting `uv`'s python-introspection JSON queries (it was breaking `uv sync` and `uv run`). Reason: `_epm_157_compat.pth` triggers `apply()` at every Python startup; vLLM logs to stdout on import; `uv` parses stdout as JSON. ~5 LOC, no logic change. The patch was previously authored on-pod by an earlier experimenter and is not tracked in the repo.
    - Reinstalled `uv` to `/root/.local/bin/uv` (lost on stop/resume; venv itself was preserved). Used `export PATH=/root/.local/bin:$PATH` in launch command.
    
    **Wall-time projection (revised from brief's ~3.7h):**
    Smoke test took 7.6 min for one source, not the brief's 3 min estimate. Updated:
    - Phase 0a (force-serial, 6 remaining sources × ~7.6 min) ≈ 46 min
    - Phase 0 (70 marker LoRAs / 8 GPUs ≈ 9 batches × 12 min) ≈ 110 min
    - Phase 0.5 (77 leakage states / 8 GPUs) ≈ 60 min
    - Phase 1 (71 JS sweep states / 8 GPUs) ≈ 45 min
    - Aggregate + plot ≈ 5 min
    - **Total ≈ 4.4 h** (ETA ~10:50 UTC)
    
    **Monitoring schedule:**
    - t+30s, t+5min, t+20min: tail log + verify Phase 0a progressing
    - t+30min, t+1h: watch for OOMs across the 8 GPUs once Phase 0 starts (R3-style failure check)
    - t+2h, t+3h, t+4h: check `result.json` counts across phases
    - On completion: aggregator + plotter + sanity 6.1#4 + downloads + `epm:results v1`
    

28. epm:experiment-implementation2026-05-05T18:22:43.000Z· system
    <!-- epm:experiment-implementation v6 --> ## Implementation Report — round 6 **Status:** READY-FOR-REVIEW **Branch:** `…

    <!-- epm:experiment-implementation v6 -->
    ## Implementation Report — round 6
    
    **Status:** READY-FOR-REVIEW
    **Branch:** `issue-228` @ `8aab236`
    **Diff vs `37720da` (R5):** +1,609 / -213 across 6 files (one new file)
    
    ### Files changed
    - `scripts/train_marker_loras_228.py` — added `rewrite_adapter_readme_base_model()`, `upload_marker_adapter_with_strict_fallback()`; refactored `train_one_state()` to disable inline `train_lora` HF push, own the upload pipeline end-to-end, hard-fail when both stores fail, and wrap merge/train/upload in `try/finally` for guaranteed local-disk cleanup.
    - `scripts/measure_leakage_228.py` — wrapped `measure_one_state` body in `try/finally` so the per-state merged dir is torn down on exception (R6 disk-hygiene fix).
    - `scripts/compute_js_convergence_228.py` — same `try/finally` wrap on `run_state` body; `is_base` short-circuit preserved.
    - `scripts/run_issue228_sweep.py` — added `_disk_free_gb`, `_check_prelaunch_disk` (>=50 GB on `/workspace` required), `_disk_monitor_thread` (warn at 50 GB, clean-shutdown at 20 GB), `_clean_orphaned_phase_scratch` (sweep stale dirs between phases). New CLI flags `--skip-disk-check` and `--min-free-gb`.
    - `scripts/salvage_marker_loras_from_wandb_228.py` — **new file**, single-process salvage script that pulls the 70 R5-trained marker LoRAs from WandB Artifacts and re-pushes to HF Hub with `base_model` rewritten. Idempotent on existing HF Hub state.
    - `tests/test_issue228_phase0_phase05.py` — 20 new R6 tests covering README rewrite (4), strict-fallback upload (4), disk-hygiene helpers (5), idempotent skip regression (2), salvage-script enumeration + state machine (5).
    
    ### Plan adherence
    
    | Brief item | Status |
    |---|---|
    | Fix #1: README `base_model` rewrite via YAML manipulation (not regex) | DONE — `yaml.safe_load` / `yaml.safe_dump` round-trip in `rewrite_adapter_readme_base_model`; rewrites field, preserves library_name / tags / body. |
    | Fix #1: unit test for README rewrite | DONE — 4 tests: replaces local-path, idempotent, no-op-when-missing, no-op-when-no-frontmatter. |
    | Fix #2: hard-fail when both HF and WandB upload fail | DONE — `train_one_state` raises `RuntimeError` on `(False, False)`; preserves local adapter dir for retry. |
    | Fix #2: HF fail + WandB ok → log WARNING, report success | DONE — `TRAINED_WANDB_ONLY` return code with explicit log: "TRAINED + WandB-uploaded (HF-upload-failed)". |
    | Fix #2: log message accurate (no false "uploaded" claim) | DONE — `TRAINED + HF-uploaded (wandb=ok|fail, ...)` on success; `TRAINED + WandB-uploaded (HF-upload-failed, ...)` on partial. |
    | Fix #3: salvage script with idempotency, sanity checks, README rewrite, HF push | DONE — `salvage_marker_loras_from_wandb_228.py` covers all 6 steps in the brief; per-state failure isolated; per-state status reported in JSON summary + log line. |
    | Fix #3: continue on per-state failure | DONE — each state runs independently in `salvage_one_state`; recorded statuses are `ALREADY_ON_HF` / `SALVAGED` / `MISSING_FROM_WANDB` / `WANDB_ARTIFACT_INCOMPLETE` / `PUSH_FAILED`. |
    | Fix #3: summary reporting | DONE — JSON summary file + one-line `SALVAGE-SUMMARY` log; exit code 2 if any holes. |
    | Fix #3: runs on local VM (no GPU) | DONE — pure WandB pull → README rewrite → HF push pipeline; no torch/vllm/transformers imports beyond what HF/WandB SDKs use. |
    | Fix #4: pre-launch disk check (<50 GB free → abort) | DONE — `_check_prelaunch_disk` raises `SystemExit` on shortfall; `--skip-disk-check` for off-pod testing. |
    | Fix #4: per-worker `try/finally` cleanup of `/workspace/tmp/issue228*` | DONE — Phase 0 (`train_marker_loras_228.train_one_state`), Phase 0.5 (`measure_leakage_228.measure_one_state`), Phase 1 (`compute_js_convergence_228.run_state`). One exception: Phase 0 keeps the adapter dir if BOTH uploads fail so retry doesn't lose ~10 GPU-min of work. |
    | Fix #4: periodic disk-monitor in coordinator | DONE — `_disk_monitor_thread` daemon thread; warns at 50 GB, sets shutdown flag at 2

29. epm:code-review2026-05-05T18:31:38.000Z· system
    <!-- epm:code-review v6 --> ## Code-Reviewer Verdict — CONCERNS **Diff size:** +1,609 / -213 across 6 files (1 new file…

    <!-- epm:code-review v6 -->
    ## Code-Reviewer Verdict — CONCERNS
    
    **Diff size:** +1,609 / -213 across 6 files (1 new file)
    **Plan adherence:** COMPLETE — all 5 R6 fixes implemented
    **Tests:** 59/59 PASS (locally reproduced from R6 working tree)
    **Lint:** **FAIL** — `scripts/salvage_marker_loras_from_wandb_228.py:167-168` has an I001 (import order) violation. The implementer's marker claims "All checks passed!" but ruff disagrees on this branch. One-line fix: alphabetize the `import wandb` / `from wandb.errors import CommError` block.
    **Format:** PASS (6 files already formatted)
    **Security sweep:** CLEAN (no hardcoded secrets, no `try/except: pass`, no shell injection vectors)
    
    ## Plan adherence — all 5 fixes verified
    
    | Fix | Status | Notes |
    |---|---|---|
    | #1 README rewrite (YAML, not regex) | ✓ | `rewrite_adapter_readme_base_model` uses `yaml.safe_load` / `yaml.safe_dump`. 4 unit tests cover replace / idempotent / no-readme / no-frontmatter. Called BEFORE `upload_model` in both train path and salvage path. |
    | #2 hard-fail on dual-store upload failure | ✓ | `train_one_state` raises `RuntimeError` on `(False, False)`; preserves `adapter_dir` for retry; logs accurate `TRAINED + HF-uploaded` / `TRAINED + WandB-uploaded (HF-upload-failed)` / `BOTH ... FAILED` messages. The R5 lying log line is fixed. |
    | #3 salvage script | ✓ | Idempotent on HF Hub (skips ALREADY_ON_HF), tempdir-isolated WandB pulls, sanity-checks adapter files, rewrites README before push, post-upload verification via `list_repo_files`. Per-state failure does not kill the loop. JSON summary + `SALVAGE-SUMMARY` log line. Exit code 2 on any holes. CPU-only (no torch/vllm/transformers imports beyond HF/WandB SDKs). |
    | #4 disk hygiene | ✓ | Pre-launch `_check_prelaunch_disk` (>=50 GB or `SystemExit`); in-flight `_disk_monitor_thread` (warn 50 / abort 20); per-worker `try/finally` cleanup in Phase 0 (`train_marker_loras_228.train_one_state`), Phase 0.5 (`measure_leakage_228.measure_one_state`), Phase 1 (`compute_js_convergence_228.run_state`); orphan-sweep between phases via `_clean_orphaned_phase_scratch`. |
    | #5 idempotent skip preserved | ✓ | Phase 0: HF Hub list-based (`_hf_already_has_marker_lora`). Phase 0.5: `marker_eval.json` skip BEFORE the new try/finally wrapper. Phase 1: `result.json` skip BEFORE the new try/finally wrapper. Two regression tests confirm. |
    
    ## Issues Found
    
    ### Critical — none.
    
    ### Major
    
    - **`scripts/salvage_marker_loras_from_wandb_228.py:167-168` lint failure (I001).** Implementer's marker says lint passed; my run on the same branch shows `Found 1 error` (`Import block is un-sorted or un-formatted`). Trivially fixable with `uv run ruff check --fix scripts/salvage_marker_loras_from_wandb_228.py`. The fact that the implementer reported PASS despite this is a credibility hit — but the actual fix is one line.
    
    ### Minor / observations
    
    - **WandB entity resolution is the single most fragile assumption in the salvage script.** `_resolve_wandb_entity()` calls `wandb.Api().default_entity`. If the user's local WandB default-entity differs from the entity the R5 worker logged the artifact under, every salvage will report `MISSING_FROM_WANDB`. The R6 marker's stated assumption ("artifact reference was logged as `wandb://huggingface/...`") is actually a misreading — `upload_model_wandb` returns `wandb://{project}/{name}:latest` (no entity field), so the R5 ref string is just `wandb://issue228_marker_loras/cp_marker_villain_ep400_s42-checkpoint:latest`, which gives zero direct evidence about the entity. The mitigation is the `--wandb-entity` override + the script's idempotency on partial success; recommend running the salvage with `--dry-run` first against `--only-source villain --only-step 200` and confirming the WandB pull works for one state before launching the full 70.
    - **`_phase0a_state_done` returns True on a non-empty file but does not parse the JSON to confirm it's a dict.** The `_ensure_completions_cache` call later fails-loud on malfo

30. epm:progress2026-05-05T19:13:51.000Z· system
    <!-- epm:progress v4 --> **Sweep relaunched after R6 hot-fix + salvage complete.** ### Salvage results (local, complet…

    <!-- epm:progress v4 -->
    
    **Sweep relaunched after R6 hot-fix + salvage complete.**
    
    ### Salvage results (local, completed 19:09 UTC)
    - 69 SALVAGED + 1 ALREADY_ON_HF = 70/70 marker LoRAs reconstructed from WandB
    - 0 holes (no MISSING_FROM_WANDB / WANDB_ARTIFACT_INCOMPLETE / PUSH_FAILED)
    - HF Hub now hosts **77 marker LoRAs** total (7 ep0 + 70 ep>0). Verified.
    
    ### R6 hot-fix `00004ee` (≤10 LOC, no logic change)
    The salvage script imported `WANDB_PROJECT = "issue228_marker_loras"` from
    `train_marker_loras_228.py`, but the R5 training run's artifacts actually
    landed under WandB project `huggingface` because HF Trainer initialized
    `wandb.run` with its default project before our training code ran, and
    `upload_model_wandb` reused the active run. Fix: hardcode `WANDB_PROJECT
    = "huggingface"` in the salvage script with a comment.
    
    Verified end-to-end on `villain/ckpt-400`: WandB pull -> README rewrite ->
    HF Hub push -> verify, then ran the full 70-state salvage successfully.
    
    ### Sweep launch (pod `epm-issue-228`)
    - **PID**: 167785 (pod side)
    - **Log**: `/workspace/issue228_sweep_r6.log`
    - **Launched**: 2026-05-05 19:10 UTC
    - **Branch**: `issue-228` @ `00004ee`
    - **Command**: `nohup uv run python scripts/run_issue228_sweep.py --num-gpus 8 --phase all --output-dir eval_results/issue_228 --leakage-output-dir eval_results/causal_proximity/strong_convergence --seed 42`
    - (Note: brief mentioned `--js-baseline-142` flag but the sweep script doesn't accept it — flag was dropped; matches the documented invocation in the script's docstring.)
    
    ### Phase progression observed (T+2 min)
    - **Phase 0a**: 7 states skipped (caches present from R5). Done in <1s.
    - **Phase 0**: Workers performing HF Hub idempotency check and early-exit
      via `ALREADY_EXISTS`. ~12s per state per worker. Already saw villain
      ckpt-200/400/600/800/1000 OK in the first batch. 70 states / 8 workers
      ≈ 110s total.
    
    ### ETA
    - Phase 0a + Phase 0: ~2 min total (idempotent skips)
    - Phase 0.5 (leakage measurement, 77 states): the bulk of the run
    - Phase 1 (JS sweep, 71 states): 8 already done from R5, 63 to run
    - Aggregate + plot at end
    - **Estimated wall time**: ~1.5–2h
    
    ### Pre-launch sanity (verified)
    - Disk: /workspace = 154GB free
    - GPUs: all 8 at 0 MiB
    - Source caches: 7/7 present at `data/leakage_v3_onpolicy/onpolicy_cache/`
    - #142 baseline: present at `eval_results/js_divergence/divergence_matrices.json`
    - Partial Phase 0/1 outputs from R5: present (will be skipped by idempotency)
    
    Monitoring continues at standard cadence (T+5min, T+30min, T+1h, T+1.5h, T+2h).

31. epm:progress2026-05-05T19:57:16.000Z· system
    <!-- epm:progress v5 --> **Phase 0a, Phase 0, and Phase 0.5 complete. Phase 1 in progress but ETA was vastly underestim…

    <!-- epm:progress v5 -->
    
    **Phase 0a, Phase 0, and Phase 0.5 complete. Phase 1 in progress but ETA was vastly underestimated.**
    
    ### Status (T+50 min)
    - **Phase 0a (on-policy cache pre-gen)**: 7/7 skipped (caches present from R5). Done in <1s.
    - **Phase 0 (marker LoRA training)**: 70/70 completed via HF Hub idempotency early-exit (~12s/state). Done in ~5 min.
    - **Phase 0.5 (leakage measurement)**: 72/77 newly completed + 5 skipped = 77/77. Done at 19:46 UTC. WandB artifact `causal_proximity_strong_convergence_v1:latest` uploaded successfully.
    - **Phase 1 (JS sweep)**: 8/71 already complete (R5 holdovers). 63 new states started at 19:47 UTC. Currently mid-batch — first batch of 8 workers at grid 22/220.
    
    ### CRITICAL: Phase 1 wall-time discovery
    
    The brief's "ETA 1.5-2h total" was based on incorrect assumptions about Phase 1 throughput.
    
    Looking at the **R5 worker log timestamps** for `comedian_ckpt200.log` (a Phase 1 state that completed in R5):
    ```
    10:28:02 grid 22/220 (616.4s elapsed)
    10:37:17 grid 44/220 (1171.0s elapsed)
    ...
    12:05:16 grid 220/220 (6450.7s elapsed)
    ```
    **Phase 1 takes ~107 minutes per state, not ~12 minutes as I'd inferred from result-file timestamps.**
    
    The R5 result-file timestamps were misleading — they showed 8 results saved in a 12-minute window because all 8 GPU workers happened to finish around the same time after each having spent ~107 min.
    
    ### Revised Phase 1 ETA
    - 63 new states ÷ 8 GPU workers = ⌈63/8⌉ = 8 batches
    - Each batch ≈ 107 min (one worker's full state)
    - **Phase 1 wall time: ~14 hours** (8 batches × ~107 min/state)
    - **Total sweep ETA: ~14 hours from launch (19:10 UTC) → completion ~09:00 UTC May 6**
    
    ### Current state (verified)
    - Sweep PID 167785 healthy (coordinator + 8 workers running)
    - Disk oscillating 40-150GB free (acceptable; cleanup keeping pace)
    - All 8 GPUs active (~15-23GB used, varying util as workers cycle through phases)
    - No errors; no crashes; idempotency working (R5 partial-results being skipped)
    
    ### What to do
    The sweep is NOT broken — Phase 1 is just inherently slow because of the per-cell teacher-force overhead × 220 cells × 11 personas × ~2-3GB models. R5 had the same throughput; the brief's ETA was just wrong about how long that translates to.
    
    **Options:**
    1. Let it run ~14h to completion (default — sweep is fully idempotent on crash/resume).
    2. Hot-fix to reduce work: cut the JS evaluation grid (e.g., fewer questions per persona), but that requires plan/implementer changes outside the experimenter scope.
    3. Scale up to more GPUs (would need a fresh provisioning cycle).
    
    The sweep is currently writing results.json files as it progresses. It is safe to leave running unattended and check back in ~14h.
    
    ### Hot-fix log
    - `00004ee` — salvage script WandB project: `issue228_marker_loras` → `huggingface` (1-line constant fix; the R5 training run actually wrote artifacts under HF Trainer's default project, not the script's WANDB_PROJECT constant).
    
    ### Salvage results
    - 69 newly SALVAGED + 1 ALREADY_ON_HF + 0 holes = 70/70 marker LoRAs reconstructed from WandB and pushed to HF Hub.
    - HF Hub now has 77 marker LoRAs total (7 ep0 + 70 ep>0).
    
    ### Pod
    - `epm-issue-228` at 103.207.149.77:13573, sweep running on PID 167785, log at /workspace/issue228_sweep_r6.log.

32. epm:results2026-05-07T19:26:29.000Z· system
    <!-- epm:results v1 --> ## Results — sweep complete (R6 path-A) **Branch:** `issue-228` @ `00004ee` (R6 + salvage hot-f…

    <!-- epm:results v1 -->
    ## Results — sweep complete (R6 path-A)
    
    **Branch:** `issue-228` @ `00004ee` (R6 + salvage hot-fix). PR #242.
    **Pod:** `epm-issue-228` (8× H100). Sweep wall: ~17h (Phase 0a/0/0.5 = 37min via salvage idempotency; Phase 1 = ~16h, ~108 min/state × 63 states / 8 GPUs).
    **GPU-hours used:** ~150 (vs. ~25 budgeted in v3 plan; underestimated Phase 1 wall by ~5×).
    
    ### Sanity check 6.1#4: PASS
    Mean relative error vs #142 baseline = **0.0106** (well under 5% threshold). Spearman ρ on flat matrices = **0.9997** (well over 0.99 threshold). 55 overlapping persona pairs.
    
    ### Outcome cell: **Mixed — cross-sectional support; longitudinal failure**
    
    | Hypothesis | Threshold | Result | Verdict |
    |---|---|---|---|
    | H1 pooled (n=71 off-diag) | \|ρ\|>0.5 negative; Δ\|ρ\|≥0.16 vs cos_L15 | ρ_JS=**−0.653** (p=6.8e-10), cluster-CI=[−0.81, −0.23]; ρ_cosL15=**+0.444** (p=0.0001), CI=[+0.01, +0.70]; Δ\|ρ\|=0.208 (CI=[−0.03, +0.44]) | **PASS on \|ρ\|; CI on Δ\|ρ\| straddles 0** |
    | H1b1 within-target z-scored | \|ρ\|>0.4 | ρ=**−0.696** (p=1.7e-11) | PASS |
    | H1b2 source-fixed mean ρ | mean \|ρ\|>0.4 | mean=−0.382; per-source negative for 5/7 | PASS |
    | H2 raw within-source | ≥4/7 FDR-PASS negative | **2/7 PASS** (librarian, software_engineer; comedian POSITIVE; nurse near zero) | **FAIL** |
    | H2 first-diff within-source | ≥4/7 FDR-PASS negative | **0/7 PASS** | **FAIL** |
    | H3 partial ρ(JS, leak \| cos_L15) | \|ρ\|>0.4 negative; CI excludes 0 | ρ=**−0.536** (p=1.7e-6), CI=[−0.74, −0.16] | PASS |
    | H3 reverse ρ(cos_L15, leak \| JS) | descriptive | ρ=**−0.061** (p=0.62), CI=[−0.42, +0.34] (straddles 0) | "Subsumed" |
    | C1 marker-mask | Δ\|ρ\|/\|ρ\|>0.4 → marker-mediated | Δ\|ρ\|=0 (mask was no-op on these data) | NOT marker-mediated |
    | C1b marker-free subset | collapse Δ\|ρ\|>0.3 → context-mediated | Δ\|ρ\|=0 | NOT context-mediated |
    | C2 prompt-distance partial | descriptive | **SKIPPED** (sentence-transformers not installed) | n/a |
    | C3 villain-out (n=61) | both n=71 and n=61 \|ρ_JS\|>0.5; Δ\|ρ\|≥0.16 at both | TBD — see correlations.json | (verify) |
    | C4 temp=1 cross-check | descriptive | **available_but_not_recomputed** | n/a |
    | C5 entropy-controlled partial | descriptive | TBD — see correlations.json | (verify) |
    
    ### H2 per-source raw
    | Source | ρ_JS | p_JS | ρ_cosL20 | sign |
    |---|---|---|---|---|
    | comedian | +0.927 | 0.0001 | +0.103 | **wrong direction** |
    | kindergarten_teacher | −0.394 | 0.26 | +0.745 | n.s. |
    | librarian | **−0.903** | 0.0003 | +0.782 | PASS |
    | medical_doctor | −0.297 | 0.41 | +0.467 | n.s. |
    | nurse | +0.139 | 0.70 | +0.261 | near zero |
    | software_engineer | **−0.745** | 0.013 | +0.879 | PASS |
    | villain | −0.588 | 0.07 | +0.527 | trend, n.s. |
    
    ### Artifacts
    - **WandB run URLs:** sweep coordinator did not attach a WandB run; per-state Phase 0 marker-LoRA training runs at `wandb://huggingface/cp_marker_<src>_ep<step>_s42-checkpoint:latest` (70 entries) + the salvage-induced WandB Artifact `causal_proximity_strong_convergence_v1` per-source leakage cache.
    - **HF Hub:** 77 marker LoRAs at `superkaiba1/explore-persona-space/adapters/cp_marker_<src>_ep<step>_s42` (7 ep0 + 70 newly-salvaged from WandB).
    - **Local artifacts (committed to issue-228 branch):**
      - `eval_results/issue_228/all_results.json` — 71 H1 cells, 70 H2 rows
      - `eval_results/issue_228/correlations.json` — full H1/H2/H3/Cn battery with cluster-bootstrap CIs
      - `figures/issue_228/{js_vs_cosine_post_convergence_hero, within_source_js_vs_leakage, within_source_rho_comparison, js_trajectories}.{png,pdf,meta.json}`
    
    ### Hot-fix log
    - `356b6b5` — PYTHONHASHSEED=42 in worker env (R3 race-condition).
    - `37720da` — drop inner FileLock (R4 deadlock fix).
    - `00004ee` — WandB project name in salvage script (R6 default-entity bug).
    - On-pod env patch: `_epm_157_compat.py` wrapped in `redirect_stdout(devnull)` (vLLM import logs corrupting uv JSON introspection).
    
    ### Standing caveats
    1. **Cross-state correlation** (JS state ≠ leakage state by 

33. epm:upload-verification2026-05-07T19:30:57.000Z· system
    <!-- epm:upload-verification v1 --> ## Upload Verification **Verdict: PASS** | Artifact | Required? | Status | URL / L…

    <!-- epm:upload-verification v1 -->
    ## Upload Verification
    
    **Verdict: PASS**
    
    | Artifact | Required? | Status | URL / Location |
    |----------|-----------|--------|----------------|
    | HF Hub marker LoRAs (77 total) | Yes | PASS | https://huggingface.co/superkaiba1/explore-persona-space (77 unique adapters: 7 ep0 + 70 ep>0, confirmed via HfApi) |
    | WandB marker LoRA checkpoints (70 × cp_marker) | Yes | PASS | `thomasjiralerspong/huggingface` project — 70 `cp_marker_*-checkpoint` model artifact collections confirmed |
    | WandB leakage cache artifact (`causal_proximity_strong_convergence_v1`) | Yes | PASS | https://wandb.ai/thomasjiralerspong/issue228 — artifact type=eval-results, size=257 MB |
    | `eval_results/issue_228/all_results.json` | Yes | PASS | Committed to `origin/issue-228` @ `ad972db` |
    | `eval_results/issue_228/correlations.json` | Yes | PASS | Committed to `origin/issue-228` @ `ad972db` |
    | `figures/issue_228/js_vs_cosine_post_convergence_hero.{png,pdf,meta.json}` | Yes | PASS | Committed to `origin/issue-228` @ `ad972db` |
    | `figures/issue_228/within_source_js_vs_leakage.{png,pdf,meta.json}` | Yes | PASS | Committed to `origin/issue-228` @ `ad972db` |
    | `figures/issue_228/within_source_rho_comparison.{png,pdf,meta.json}` | Yes | PASS | Committed to `origin/issue-228` @ `ad972db` |
    | `figures/issue_228/js_trajectories.{png,pdf,meta.json}` | Yes | PASS | Committed to `origin/issue-228` @ `ad972db` |
    | Local pod weights cleaned (`/workspace/tmp`) | Yes | PASS | `/workspace/tmp` = 148 KB, 0 safetensors files remaining |
    | Pod lifecycle | Yes | WARN | Pod `epm-issue-228` still **running** (status: running, 8×H100). Should be stopped after upload verification is complete. |
    
    **Missing:** None
    
    **Notes:**
    - The 7 ep0 adapters exist on HF Hub but have no corresponding WandB checkpoint artifacts — this is expected since ep0 adapters are the untrained baseline (never went through a WandB-instrumented training run). The plan requires WandB artifacts only for ep>0 marker LoRAs; all 70 are present.
    - The leakage cache artifact was uploaded to WandB project `issue228` (not `huggingface`) — this is the correct project; the upload function uses `project="issue228"` by design.
    - The ep0 adapters on HF Hub were salvaged from WandB (they are the base LoRA state before convergence training), not independently trained, so no separate WandB training artifact is expected.
    - Pod warning: `epm-issue-228` is still running. Per lifecycle rules, it should be stopped now that upload verification has passed. Run: `uv run python scripts/pod.py stop --issue 228`
    <!-- /epm:upload-verification -->
    

34. epm:original-body2026-05-07T19:51:10.000Z· system
    <!-- epm:original-body v1 --> ## Goal Understand whether JS divergence (output-space metric) explains the convergence t…

    <!-- epm:original-body v1 -->
    ## Goal
    
    Understand whether JS divergence (output-space metric) explains the convergence training results better than cosine similarity (representation-space metric). Specifically:
    
    1. **Does JS divergence predict marker leakage post-convergence?** Issue #142 showed JS divergence predicts base-model leakage (rho=-0.75) better than cosine (rho=0.57), but only on the unfinetuned model. This issue extends the analysis to each checkpoint of the convergence training experiments.
    
    2. **Does ΔJS divergence predict Δmarker leakage within-source?** Issue #91/#109 showed cosine similarity does NOT predict leakage within-source (leakage DECREASES as cosine increases). Does JS divergence do better?
    
    3. **Cross-metric rationalization.** How much of the variation in cosine similarity, JS divergence, and marker leakage across convergence checkpoints can be explained by a simple correlation structure?
    
    ## Data Source
    
    - **Issue #109** — 7 source personas × 11 convergence checkpoints (epoch 0, 2, 4, ..., 20) × 11 evaluation personas × 20 questions × 10 completions. Marker leakage and cosine similarity already computed. Adapters on HF Hub.
    
    ## Method
    
    - For each of 77 (source × checkpoint) adapters: merge with base, generate 220 responses (11 personas × 20 questions) via vLLM, teacher-force each response under all 11 system prompts, compute per-pair JS divergence (same protocol as #142).
    - Correlate: JS vs leakage at each checkpoint (cross-source), ΔJS vs Δleakage within-source, partial correlations controlling for cosine.
    - Compare JS predictive power post-convergence vs baseline (#142).
    
    ## Hypothesis
    
    JS divergence will predict marker leakage post-convergence better than cosine similarity does, and ΔJS will correlate with Δleakage within each source persona's convergence trajectory. The three behavioral groups identified in #109 (high: villain/med doc; medium: comedian/KT; low: SW eng/nurse/librarian) will map onto JS divergence clusters.
    
    ## Success Criterion
    
    ΔJS ↔ Δleakage within-source Spearman rho magnitude > 0.3 (significant at p<0.05 after FDR correction across 7 sources).
    
    ## Kill Criterion
    
    JS divergence does no better than cosine at predicting leakage post-convergence — rho improvement < 0.1.
    
    ## Compute Estimate
    
    ~2-4 GPU-hours on 1× H100 (77 forward-pass batches of 220 responses × 11 teacher-forced passes each; mostly inference, no training).
    
    ## Parent
    
    Parent: #109 (convergence SFT results), builds on #142 (JS divergence baseline)
    
    <!-- /epm:original-body -->
    

35. epm:analysis2026-05-07T19:51:29.000Z· system
    <!-- epm:analysis v1 --> ## Analyzer output (round 1) - **Clean-result issue:** #228 (this issue body has been replaced…

    <!-- epm:analysis v1 -->
    ## Analyzer output (round 1)
    
    - **Clean-result issue:** #228 (this issue body has been replaced with the polished clean-result; original body preserved in [the previous comment](https://github.com/superkaiba/explore-persona-space/issues/228#issuecomment-4400565580)).
    - **Hero figure:** https://raw.githubusercontent.com/superkaiba/explore-persona-space/ad972db/figures/issue_228/js_vs_cosine_post_convergence_hero.png
    - **Confidence verdict:** **MODERATE**.
    
    **Recap (2 sentences):** JS divergence predicts post-convergence marker leakage cross-sectionally at rho=-0.65 (p=6.8e-10, n=71) and subsumes cosine_L15 in the H3 partial-correlation test (rho_partial(JS|cos_L15)=-0.54 vs reverse=-0.06), replicating #142's base-model finding on convergence-trained checkpoints. The within-source longitudinal H2 fails decisively (2/7 raw FDR-PASS, 0/7 first-difference, comedian inverts to rho=+0.93), so the metric captures the post-convergence equilibrium across sources/targets but does not track within-trajectory training dynamics — outcome cell is "Mixed: cross-sectional support; longitudinal failure."
    
    Verifier: PASS (1 numeric-WARN — 16 cited numbers are sample-output cell values from `all_results.json` which the verifier only scans against `correlations.json`).
    <!-- /epm:analysis -->
    

36. epm:interp-critique2026-05-07T19:56:03.000Z· system
    <!-- epm:interp-critique v1 --> ## Interpretation Critique — Round 1 **Verdict: REVISE** The cross-sectional + longitu…

    <!-- epm:interp-critique v1 -->
    ## Interpretation Critique — Round 1
    
    **Verdict: REVISE**
    
    The cross-sectional + longitudinal framing in the title is honest and the MODERATE confidence call is well-placed. However, there are several specific issues — one of which (the C1/C1b "no-op" claim and one factually-wrong "Next steps" line) materially affects the calibration and must be fixed before promotion. Most others are surfaced-but-could-be-clearer.
    
    ### Overclaims
    
    - **"JS subsumes cosine" framing in takeaway #2 is too strong given the Δ|ρ| CI straddles 0.** Takeaway #3 already concedes this for the headline Δ|ρ|=0.21 number, but the word "subsumes" in takeaway #2 is the same overclaim that #142 was warned against in plan §3 ("'subsumed' is overclaimed shorthand; report partial-ρ + p and let the CI speak"). The partial-correlation point estimate is clean (ρ=−0.54 vs ρ=−0.06), but the report still inherits a single-seed n=71 design with a 7-cluster cluster-bootstrap CI. Suggested weakening: "the linear effect of cosine_L15 on leakage is largely absorbed once JS is held constant" rather than "JS subsumes cosine".
    
    - **Takeaway #4 calls H2's failure "a clean falsification of the within-trajectory framing of #91/#109".** This is overstated. H2 raw passed 2/7 (≥4/7 was the threshold), and 4/7 sources had negative JS-leakage ρ point estimates (KT −0.39, librarian −0.90, medical_doctor −0.30, software_engineer −0.75, villain −0.59 → 5/7 negative). The threshold was missed but the directional pattern is mostly consistent with H1; the falsified thing is "JS tracks within-source dynamics with the strength needed to clear FDR", not "the within-trajectory framing". Suggested weakening: "H2 fails to clear the pre-registered ≥4/7 FDR-PASS bar (only 2/7 raw + 0/7 first-difference); 5/7 sources are negative in sign but most miss FDR, and comedian inverts to ρ=+0.93".
    
    - **C1 / C1b "the marker mask was a no-op on these data" (table footer) is true but underspecified.** Independently inspecting `all_results.json`, `js`, `js_masked`, and `js_no_marker` are bit-identical in every one of the 71 cells. That is suspicious enough to warrant a one-line note: either (a) the teacher-forced responses contain zero `[ZLT]` tokens (plausible since these are convergence-only adapters, not marker-LoRA states), or (b) the mask code never triggered. The current phrasing ("NOT marker-mediated") could be misread as "we tested the mask and confirmed marker doesn't drive the signal" when in fact the mask had no datapoints to act on. Suggested addition to the standing caveats: "C1/C1b were no-ops because js_masked == js_no_marker == js bit-identically across all 71 cells — consistent with the convergence-only adapter teacher-forcing zero `[ZLT]` tokens; this means the marker-confound check is uninformative on these data, NOT that we ruled it out."
    
    ### Surprising Unmentioned Patterns
    
    - **Within-source cosine_L20 is POSITIVE for 7/7 sources in H2 raw** (KT +0.75, librarian +0.78, software_engineer +0.88, comedian +0.10, medical_doctor +0.47, nurse +0.26, villain +0.53). This is the OPPOSITE direction from #91's "cosine increases → leakage decreases" within-source finding. The body acknowledges H2-JS fails the bar but doesn't surface that **cosine_L20 within-source is positive for ALL 7 sources** — which is the cleaner, more interesting falsification to highlight (it directly inverts #91 within the same data). Suggest adding a takeaway-#4 footnote: "Within-source cosine_L20 also flips: ρ>0 for all 7 sources (n=10 each), reversing #91's negative-sign within-source finding — both metrics fail to track #91's direction longitudinally on this dataset."
    
    - **Most n=7 within-source JS rhos are NEGATIVE in sign (5/7), even where FDR fails.** Only comedian and nurse are positive. The body's takeaway #4 names comedian's positive flip but doesn't say that 5/7 sources are negative. This understates the directional consistency with H1.
    
    - **C5 entropy-controlled partial CI is [-0.7

37. epm:interpretation2026-05-07T20:07:30.000Z· system
    <!-- epm:interpretation v2 --> ## Interpretation Round 2 — Revisions from R1 critique R1 critic verdict: **REVISE**. Th…

    <!-- epm:interpretation v2 -->
    ## Interpretation Round 2 — Revisions from R1 critique
    
    R1 critic verdict: **REVISE**. The following changes were applied to the body before re-promotion:
    
    - **B1 (blocking) — C1/C1b reframed as UNINFORMATIVE, not confirmed nulls.** Independently verified: `js_masked == js_no_marker == js` bit-identically across all 71 cells in `eval_results/issue_228/correlations.json`, because the convergence-only adapter teacher-forced (temp=0 greedy) responses contain zero `[ZLT]` substrings — so the mask had no datapoints to act on. The body's headline-numbers table now reads "UNINFORMATIVE (no-op on this dataset)" for both cells, the standing caveats explicitly flag "we did NOT rule out marker-mediation; we showed only that the mask was a no-op on temp=0 responses", and a new Next-steps bullet proposes re-running C1/C1b on temp=1 sampled responses where markers actually appear.
    
    - **B2 (blocking) — Next-steps software_engineer error fixed.** v1 named software_engineer as a within-source H2 failure case; software_engineer is in fact one of the **2/7 H2-raw-FDR PASS** sources (rho_JS=-0.745, p_adj=0.031). The corrected Next-steps line names the actual failure / null cases — comedian (the +0.93 inversion), kindergarten_teacher (rho=-0.39), nurse (+0.14), medical_doctor (-0.30), villain (-0.59) — and contrasts them with librarian (-0.90) and software_engineer (-0.75) as the H2-PASS pair.
    
    - **T1 — "JS subsumes cosine" softened in takeaway #2.** v1 said "JS subsumes cosine_L15's predictive power"; per `correlations.json`, the paired Delta|rho| comparison itself has CI=[-0.026, +0.441] that straddles 0, so we cannot reject "equally good predictors" from that margin alone. v2 reframes: "JS-as-predictor is consistent with subsuming cosine_L15's signal" — the partial rho(JS | cos_L15)=-0.54 (CI=[-0.74, -0.16] excludes 0) is the load-bearing evidence, NOT the Delta|rho| beat.
    
    - **T2 — H2 "clean falsification" softened in takeaway #4.** v1 called H2's failure "a clean falsification of the within-trajectory framing of #91/#109"; in fact 5/7 within-source rho_JS are negative in sign (only the FDR threshold misses). v2 reframes as "the pre-registered FDR bar fails — but the directional pattern is mostly consistent with H1: 5/7 sources are negative, comedian inverts to +0.93" — calibrated as "underpowered consistency" rather than "absence of effect".
    
    - **T3 — New takeaway on within-source cosine_L20 sign pattern.** Per `correlations.json` `h2.per_source_raw`, within-source rho_cos_L20 is POSITIVE for 7/7 sources (range +0.10 to +0.88, mean ~+0.55) — partially replicating #91's positive within-source cosine-vs-leakage trend. New takeaway in TL;DR Results: JS indexes the cross-state equilibrium structure while cosine_L20 indexes the per-source training trajectory — the metrics capture different layers of the same phenomenon rather than competing for the same explanatory role. Also added as a descriptive row in the Headline numbers table.
    
    - **T4 — C5 entropy-controlled CI nuance noted.** v1 reported "passes" without flagging that the CI lower bound nearly touches 0; per `correlations.json` `c5_entropy_partial`, rho=-0.566, p=4.0e-7, n=71, **CI=[-0.74, -0.01]**. v2 standing caveats now state explicitly: "the per-state output-entropy covariate absorbs a meaningful but not dominant fraction of the JS signal; the JS effect is mostly NOT explained by per-state entropy collapse, modulo a small fraction at the threshold." Headline-numbers table now also shows "CI=[-0.74, -0.01]; lower bound nearly touches 0".
    
    Title and confidence unchanged: **MODERATE** confidence stays correctly placed (per critic's calibration verdict). Confidence sentence updated to name the new binding constraints explicitly.
    
    Verifier: PASS (`scripts/verify_clean_result.py` — all checks pass; the 16-numeric-claim WARN is sample-output / table-cell numbers not present in `correlations.json`, carried over from v1).
    <!-- /epm:interpretation -->
    

38. epm:interp-critique2026-05-07T20:09:51.000Z· system
    <!-- epm:interp-critique v2 --> ## Interpretation Critique — Round 2 **Verdict: PASS** All six R1 revision requests ar…

    <!-- epm:interp-critique v2 -->
    ## Interpretation Critique — Round 2
    
    **Verdict: PASS**
    
    All six R1 revision requests are addressed in v2 with adequate fidelity to the raw data. Independent re-verification against `eval_results/issue_228/correlations.json` confirms every load-bearing number in the v2 body matches the JSON.
    
    ### B1 (blocking) — RESOLVED
    v2 reframes C1/C1b as UNINFORMATIVE in three places: (1) the headline table shows "UNINFORMATIVE (no-op on this dataset; see standing caveats)" for both rows; (2) standing caveat #3 explicitly states "we did NOT rule out marker-mediation or context-mediation; we showed only that the mask was a no-op on temp=0 responses"; (3) the Next-steps bullet proposes re-running on temp=1 sampled responses where markers actually appear. This is the correct framing — the JSON shows `delta_abs_rho: 0.0` because `js_masked == js_no_marker == js` bit-identically, not because marker-mediation was tested-and-rejected.
    
    ### B2 (blocking) — RESOLVED
    v2 Next-steps correctly names software_engineer as one of the two H2-raw-FDR-PASS sources (alongside librarian), and the failure/null cases as comedian (+0.93), kindergarten_teacher (-0.39), nurse (+0.14), medical_doctor (-0.30), villain (-0.59). Cross-checked against `correlations.json` `raw_fdr`: software_engineer p_adj=0.031, fdr_pass=true, negative_sign=true. The factual error is gone.
    
    ### T1 — RESOLVED
    Takeaway #2 now reads "JS-as-predictor is consistent with subsuming cosine_L15's signal" (was "JS subsumes cosine_L15's predictive power") and explicitly names the partial-correlation test (CI excluding 0) as the load-bearing evidence rather than the Delta|rho| margin (which has CI=[-0.026, +0.441] straddling 0). The "subsumes" word is gone from the takeaway and now appears only inside the qualified clause "is consistent with subsuming" — acceptable.
    
    ### T2 — RESOLVED
    Takeaway #3 (was #4) replaces "clean falsification" with "the pre-registered FDR bar fails — but the directional pattern is mostly consistent with H1: 5/7 within-source rho_JS values are negative in sign". This frames the H2 failure as "underpowered consistency" rather than "absence of effect", which is what the data supports. Verified 5/7 negative from `raw_fdr`: KT, librarian, MD, SE, villain all `negative_sign: true`.
    
    ### T3 — RESOLVED
    New takeaway #4 captures the cosine_L20 7/7 positive pattern with the correct numbers (range +0.10 to +0.88, mean ~+0.55). One minor framing nuance: v2 calls it "partially replicating #91's positive within-source cosine-vs-leakage trend" rather than R1's suggested "inverts #91". I'll defer to the analyzer's read of #91 here — if the within-source cosine-vs-leakage in #91 was indeed positive (not negative), then v2's framing is more accurate than my R1 suggestion. Either way the 7/7 sign pattern is now surfaced as load-bearing.
    
    ### T4 — RESOLVED
    C5 entropy partial now flagged in standing caveats with the correct numbers ("rho=-0.566, p=4.0e-7, n=71, CI=[-0.74, -0.01]") and described as "the per-state output-entropy covariate absorbs a meaningful but not dominant fraction of the JS signal". Headline table also shows "passes; lower CI bound nearly touches 0". Calibrated honestly.
    
    ### Fresh 5-lens re-pass (new issues only)
    
    **Overclaims:** None new. The "JS and cosine_L20 capture *different layers of the same phenomenon*" framing in takeaway #4 is a reasonable interpretation, not an overclaim — and it is hedged ("appears to index", "appears to index"). Pass.
    
    **Surprising unmentioned patterns:** None new. The within-source z-score robustness (rho=-0.696), villain-out C3 (rho=-0.672), source-fixed mean (-0.382, 5/7 negative), and target-fixed mean (-0.667, 10/11 negative) all appear in the headline table. The H1 source-fixed mean of |rho|=0.486 is borderline-PASS (threshold 0.4) and is correctly flagged as "borderline" in the table. Pass.
    
    **Alternative explanations:** R1 raised the "JS / leakage / cosine all downstream of training intensity" alternativ

39. epm:reviewer-verdict2026-05-07T20:14:14.000Z· system
    <!-- epm:reviewer-verdict v1 --> ## Independent Reviewer Verdict — issue #228 **Verdict:** PASS (with concerns) ### Pe…

    <!-- epm:reviewer-verdict v1 -->
    ## Independent Reviewer Verdict — issue #228
    
    **Verdict:** PASS (with concerns)
    
    ### Per-axis verdicts
    
    | Axis | Verdict | Notes |
    |---|---|---|
    | Numerical accuracy | PASS (one minor row error) | Spot-checked 12 cited values against `correlations.json` and `all_results.json`; all H1/H2/H3/C3/C5 headline numbers match exactly. Independently re-ran `spearmanr` on raw cells: rho_JS = -0.6528 (p=6.85e-10, n=71), rho_cos_L15 = +0.4445 (p=1.03e-04) — bit-identical to JSON and body. Sample-output cells (SE -> data_scientist, SE -> villain, comedian -> zelthari, etc.) all verified. Hero-figure URL is commit-pinned (`ad972db`) and HTTP-200 reachable. **One discrepancy:** Headline-numbers table row "H1b2 source-fixed ... (5/7 sources negative)" — actual is **6/7** negative (only comedian +0.367 is positive in `h1b2_source_fixed`). The "5/7 negative" figure IS correct for H2-raw (within-source), but is misapplied to the H1b2 row. |
    | Calibration | PASS | Title "predicts cross-sectionally but fails longitudinally" matches the plan's (PASS, FAIL) "Mixed: cross-sectional only" outcome cell. Confidence MODERATE matches the title. Binding constraints (single seed, Δ\|ρ\| margin straddles 0, H2 FDR fails) are explicitly enumerated in the Confidence sentence. The "JS subsumes cosine" claim is properly attributed to the partial-correlation test (CI excludes 0) rather than the Δ\|ρ\| margin (CI straddles 0) — this is the correct load-bearing evidence and the body says so out loud. |
    | Caveats completeness | PASS (unusually thorough) | Standing-caveats block covers all the bases I would have demanded: single seed=42, cross-state correlation, C1/C1b UNINFORMATIVE (not confirmed null) on temp=0 responses, Δ\|ρ\| margin tightness, [ZLT] substring FP risk inherited from #109, villain dropout asymmetry tested via C3, C2 SKIPPED (sentence-transformers missing), behavioral group labels descriptive only, effective N << 71 due to crossed dependence, C5 CI lower bound nearly touches 0, C4 not run, n=70 -> n=71 amendment, sign-flip vs #109 explained. The C1/C1b "uninformative, not confirmed null" caveat is exactly the kind of honest disclosure that often gets papered over — credit for stating it explicitly. |
    | Template compliance | PASS | TL;DR has the required 4 H3 subsections (Background, Methodology, Results, Next steps) in order. Results subsection has hero figure -> 1-2 sentence figure description with N inline -> `**Main takeaways:**` with 5 bullets bolding the load-bearing claim -> single `**Confidence: MODERATE** — ...` line. Detailed report has all required sections (Source issues, Setup & hyper-parameters with rationale prose at top, WandB, Sample outputs, Headline numbers with Standing caveats inline, Artifacts). Reproducibility card has no sentinels (`{{`, `TBD`, `see config`, `default`). `uv run python scripts/verify_clean_result.py` -> PASS. |
    | Statistical reporting | PASS (borderline on Δ-framing) | rho + p + n reported throughout; no Cohen's d / η² / `value ± err` in prose. Bootstrap-CI numbers come from cluster-resampling but are presented as `CI=[low, high]` rather than `value ± err`. **Borderline:** the body uses "Δ\|rho\|", "paired ... comparison", "BH-FDR" in prose, which a strict reading of CLAUDE.md ("no Δ-framed-as-effect ... no named statistical tests ... in prose") could flag. The verifier's `Stats framing` check accepted it, and the Δ\|ρ\| discussion is substantively necessary to honestly characterize what cleared and what did not — so I am treating this as acceptable convention-bending rather than a violation. Worth flagging in the next polish pass if the convention is to be enforced strictly. |
    
    ### Concerns (non-blocking — for future polish, do NOT trigger another revision round)
    
    1. **Headline-numbers H1b2 row arithmetic.** The text "(5/7 sources negative)" in the H1b2 source-fixed row should be "(6/7 sources negative)" — the JSON `h1b2_source_fixed.per_source` has only comedian (+0.367) posi

40. epm:awaiting-promotion2026-05-07T20:14:45.000Z· system
    <!-- epm:awaiting-promotion v1 --> ## Reviewer PASS — clean-result ready for your review `epm:reviewer-verdict v1` PASS…

    <!-- epm:awaiting-promotion v1 -->
    ## Reviewer PASS — clean-result ready for your review
    
    `epm:reviewer-verdict v1` PASS (with 4 non-blocking nits): https://github.com/superkaiba/explore-persona-space/issues/228#issuecomment-4400719281
    
    The clean-result body is at the top of this issue (replaced from the original spec; original preserved as `epm:original-body v1`). Title: `JS divergence predicts convergence-trained marker leakage cross-sectionally but fails longitudinally (MODERATE confidence)`.
    
    ### To promote
    ```
    /clean-results promote 228
    ```
    This removes the `:draft` suffix and moves the issue into the Clean Results column.
    
    ### To revise instead
    Reply with notes here, or invoke another analyzer round (will need user override beyond round 2).
    
    ### After promotion
    Re-invoke `/issue 228` to auto-complete (Step 10): updates RESULTS.md, eval_results/INDEX.md, advances to `status:done-experiment`, spawns follow-up-proposer, and asks for pod termination permission.
    <!-- /epm:awaiting-promotion -->

41. epm:clean-result-lint2026-05-08T13:03:41.000Z· system
    <!-- epm:clean-result-lint v1 --> ## Clean-result lint — FAIL ``` Check Status Detail -----…

    <!-- epm:clean-result-lint v1 -->
    ## Clean-result lint — FAIL
    
    ```
    Check                            Status  Detail
    ---------------------------------------------------------------------------------------------------
    AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
    Human TL;DR                      ✓ PASS  section missing (legacy body — grandfathered)
    AI TL;DR paragraph               ✓ PASS  430 words, 5 bullets (LW-style)
    Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
    Results figure captions          ✓ PASS  every Results figure has a caption paragraph
    Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
    Methodology bullets              ✓ PASS  pre-cutoff (created 2026-05-04, cutoff 2026-05-15)
    Background context               ✓ PASS  Background has 90 words
    Acronyms defined                 ✓ PASS  all defined: ['H1', 'H2', 'H3']
    Background motivation            ✓ PASS  references prior issue(s): [109]
    Dataset example                  ✓ PASS  dataset example + full-data link present
    Human summary                    ✗ FAIL  ## Human summary section missing (must appear at top of Detailed report)
    Sample outputs                   ✗ FAIL  ## Sample outputs section missing
    Numbers match JSON               ✓ PASS  no JSON artifacts referenced — skipped
    Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
    Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
    Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
    Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match
    
    Result: FAIL — fix the failing checks before posting.
    ```
    
    Fix the issues and edit the body; the workflow re-runs.
    <!-- /epm:clean-result-lint -->

42. epm:clean-result-lint2026-05-08T14:52:27.000Z· system
    <!-- epm:clean-result-lint v1 --> ## Clean-result lint — FAIL ``` Check Status Detail -----…

    <!-- epm:clean-result-lint v1 -->
    ## Clean-result lint — FAIL
    
    ```
    Check                            Status  Detail
    ---------------------------------------------------------------------------------------------------
    AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
    Human TL;DR                      ✓ PASS  section missing (legacy body — grandfathered)
    AI TL;DR paragraph               ✓ PASS  396 words, 5 bullets (LW-style)
    Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
    Results figure captions          ✓ PASS  every Results figure has a caption paragraph
    Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
    Methodology bullets              ✓ PASS  pre-cutoff (created 2026-05-04, cutoff 2026-05-15)
    Background context               ✓ PASS  Background has 98 words
    Acronyms defined                 ✓ PASS  no project-internal acronyms used
    Background motivation            ✓ PASS  references prior issue(s): [109]
    Bare #N references               ✓ PASS  skipped (v1 / legacy body — markdown-link rule applies to v2 only)
    Dataset example                  ✓ PASS  dataset example + full-data link present
    Human summary                    ✗ FAIL  ## Human summary section missing (must appear at top of Detailed report)
    Sample outputs                   ✗ FAIL  ## Sample outputs section missing
    Inline samples per Result        ✓ PASS  2 Result section(s), each with >=2 fenced sample blocks
    Numbers match JSON               ✓ PASS  no JSON artifacts referenced — skipped
    Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
    Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
    Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
    Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match
    
    Result: FAIL — fix the failing checks before posting.
    ```
    
    Fix the issues and edit the body; the workflow re-runs.
    <!-- /epm:clean-result-lint -->

43. epm:clean-result-lint2026-05-09T04:37:41.000Z· system
    <!-- epm:clean-result-lint v1 --> ## Clean-result lint — FAIL ``` Check Status Detail -----…

    <!-- epm:clean-result-lint v1 -->
    ## Clean-result lint — FAIL
    
    ```
    Check                            Status  Detail
    ---------------------------------------------------------------------------------------------------
    AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
    Human TL;DR                      ✓ PASS  H2 present (content user-owned, not validated)
    AI TL;DR paragraph               ✓ PASS  396 words, 5 bullets (LW-style)
    Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
    Results figure captions          ✓ PASS  every Results figure has a caption paragraph
    Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
    Methodology bullets              ✓ PASS  pre-cutoff (created 2026-05-04, cutoff 2026-05-15)
    Background context               ✓ PASS  Background has 98 words
    Acronyms defined                 ✓ PASS  no project-internal acronyms used
    Background motivation            ✓ PASS  references prior issue(s): [109]
    Bare #N references               ✓ PASS  skipped (v1 / legacy body — markdown-link rule applies to v2 only)
    Dataset example                  ✓ PASS  dataset example + full-data link present
    Human summary                    ✗ FAIL  ## Human summary section missing (must appear at top of Detailed report)
    Sample outputs                   ✗ FAIL  ## Sample outputs section missing
    Inline samples per Result        ✓ PASS  2 Result section(s), each with >=2 fenced sample blocks
    Numbers match JSON               ✓ PASS  no JSON artifacts referenced — skipped
    Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
    Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
    Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
    Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match
    
    Result: FAIL — fix the failing checks before posting.
    ```
    
    Fix the issues and edit the body; the workflow re-runs.
    <!-- /epm:clean-result-lint -->

44. state_changed2026-05-13T22:33:13.894Z· user· awaiting_promotion → reviewing
    Bulk move clean-results → review (kept #311 in clean-results)

    Bulk move clean-results → review (kept #311 in clean-results)

45. state_changed2026-05-14T00:23:27.541Z· user· reviewing → archived
    Superseded by lead #207 — clean result combined cluster B (persona geometry predicts cross-persona marker leakage)

    Superseded by lead #207 — clean result combined cluster B (persona geometry predicts cross-persona marker leakage)

Comments · 0

No comments yet. (Auth + comment composer land in step 5.)