EPS
← All docs·Activity

Daily — 2026-06-02 (6 experiments parked at awaiting_promotion; 9 workflow fixes auto-landed)

updated: 2026-06-03

What happened

  • Six experiments reached a clean-result and parked at awaiting_promotion today: #444 (four fact-teaching leakage shapes + a 5-way re-judge), #464 (chat-template role-header marker localization), #466 (hypothesis-led "habit leaks in inverse proportion to how differently two personas answer"), #468 (#463 cosine→EM signal survives at the last user-content token, full 3-round ensemble PASS), #470 (JS-similarity is the worst of 13 sycophancy-leakage predictors), #472 (held-out marker leakage tracks source-implant strength, not where negatives sit). None were promoted to completed (promotion is user-only).
  • #405 promoted its body in place to a v2 clean-result (LOW confidence — K-by-distance interaction driven by one far persona), pod terminated after upload-verification PASS; sits at interpreting.
  • Four experiments still running: #467 (disentangle persona-geometry from training-question content; re-planned mid-flight around strong-NL prompt + {cosine, JS@R≥8}), #471 (rerun #465 with training-time contrastive negatives), #474 (on-policy divergence-to-transfer; hit a disk-EDQUOT infra failure, recovered), #475 (CoT-scaffolded marker install on Qwen3.5-27B; fought an infra storm — SUPPLY_CONSTRAINT, uv wiped on resume, torch cu130-vs-driver-cu128 mismatch, MooseFS errno 116).
  • Heavy in-session workflow self-improvement — 9 workflow-surface commits landed on main: grep-the-literal code-reviewer rule (after #467 r1 shipped a fabricated "R=16" PASS on R=8 code), auto-arm /loop backstop cron (no more chat reminder), don't-relitigate-prior-methodology + auto-commit/merge/push for workflow edits (both Thomas's explicit asks), contrastive-negatives rule + planner §4 / critic Methodology-lens wiring, goal auto-link (ask only for new questions), phantom-URL upload-verification gate, pod.py-terminate-guards-upload-PASS, CLAUDE.md slimmed 50k→38k.
  • Ensemble review machinery exercised hard: #464 took 4 code-review rounds, #470 had a reconciler overturn a Claude PASS to FAIL (Codex caught a silent None fill), #467 a reconciler caught the fabricated R=16 PASS. Several tasks archived: #412, #443, #453, #456, #460, #462, #476.

Proposed workflow improvements

Many of the day's highest-value fixes already auto-landed in-session (see "What happened"). The items below are the gaps that did not get fixed today. Ordered by severity. PROPOSED only — reply "do 1, 4" / "do all" / "skip, talk about 3" and workflow-improver applies the greenlit ones.

  1. Target: .claude/agents/upload-verifier.md + CLAUDE.md (Upload Policy table) — what: make training-mix datasets (JSONL) a verified artifact class, not the one upload row with no verifier coverage. Why: Session ea1897f8 [H84] "why didn't the upload verifier catch these things" + [H86] agent report: #444's hand-orchestrated completion bypassed Step 8 and silently lost the training-mix datasets. The Upload Policy table lists datasets as "Auto after generation" — the only artifact class with no verifier check, which is exactly how they vanished unflagged. Proposed edit:

    # .claude/agents/upload-verifier.md — artifact checklist
+ - [ ] Training-mix datasets (JSONL) on HF data repo — list_repo_files
+       must show each dataset the run generated; a missing dataset is a
+       hard FAIL (incident #444: hand-orchestrated completion lost the mixes).
# CLAUDE.md — Upload Policy table
- | Datasets (JSONL training mixes) | HF data repo | Auto after generation |
+ | Datasets (JSONL training mixes) | HF data repo | Auto after generation; upload-verifier confirms each via list_repo_files at Step 8 |

    Same incident also lost #470's 24 non-regenerable base-model sample JSONs (ae9d3712 [H40]); the verifier should enumerate temp=1.0 base-model response JSONs too. Confidence: high

  2. Target: .claude/agents/analyzer.md — what: add a plain-language "newcomer test" gate AND require the body to lead with the task's original registered hypothesis, not an analyzer-chosen framing. Why: Session 53909073 [H68] "What does slice-resolved JS mean? you're not supposed to use jargon like this", [H72] "Remove jargon like 'slice-resolved'. Everything should be understandable even to someone who just joined the project" AND "The main finding should be that my hypothesis was right and everything should focus around that". The #466 draft carried the plan's internal term verbatim and drifted from the registered question — forcing a full reframe pass. (The don't-relitigate-prior-issues rule landed today; the plain-language + hypothesis-led gates are separate and did not.) Proposed edit:

    + ## Plain-language gate (newcomer test) + hypothesis-led framing
+ Before drafting, list every multi-word noun phrase naming a method/metric/
+ condition that did NOT come from the field (e.g. "slice-resolved divergence",
+ "slice-aware", "always-Spanish control"). Each must be replaced with plain
+ English or glossed in-line on first use. Test: would someone who joined the
+ project today understand it without the plan? Plan shorthand NEVER enters the
+ title or Human TL;DR verbatim.
+ The title + Human TL;DR + ### Motivation frame the result as the answer to the
+ task's `goal:` / `## Goal` question (read it from frontmatter). If the result
+ confirms the stated hypothesis, the headline IS "hypothesis held", plainly.

    Confidence: high

  3. Target: .claude/agents/clean-result-critic.md — what: (a) state that a mechanical verify_task_body.py / audit PASS is necessary but NOT sufficient — re-read SPEC.md prose rules section-by-section; (b) reclassify bare opaque condition codes in Human TL;DR / Findings prose / figure legends from CONCERN to BLOCKER. Why: Recurring reconciler overturns of the Claude critic. Session 54c4425b [H15]: reconciler sided with Codex — "Claude missed [4 real SPEC-text violations] by treating mechanical pre-pass PASS as ground truth and not re-reading SPEC.md against the body" (standalone ### What I ran, opaque codes). Opaque codes (A1–A5, A3 ∪ A4) repeatedly surfaced as a soft CONCERN ([H12], 5b78eb5c [H16]) despite the standing feedback_no_opaque_condition_codes.md rule that they must NEVER appear. Proposed edit:

    + verify_task_body.py / audit PASS is NECESSARY, NOT SUFFICIENT — those checks
+ are mechanical and do NOT verify SPEC.md prose rules (standalone `### What I
+ ran`, plain-English condition names in prose, no bracketed CIs). Re-read
+ SPEC.md section-by-section against the body before any PASS.
- Lens 2: bare letter/opaque condition codes are a CONCERN.
+ Lens 2: bare letter/opaque condition codes (A1-A5, cond_4, M1, A3 ∪ A4) in
+ Human TL;DR, Findings prose, OR figure legends are a BLOCKER, not a CONCERN.

    Confidence: high

  4. Target: .claude/agents/analyzer.md (line ~489) — what: fix the stale reviewer-agent reference + outdated promotion flow. Why: analyzer.md:489 still says "The reviewer agent reads the raw data ... posts a verdict event. On PASS, the /issue skill sets status='awaiting_promotion'" — but the reviewer agent was deprecated 2026-05-13 and its role absorbed by clean-result-critic. Session 360351d6 [H41] shows the analyzer confusing itself over what advances the status. The current flow is analyzer → interpretation-critic → clean-result-critic (final gate) → orchestrator advances to awaiting_promotion. Proposed edit:

    - The `reviewer` agent reads the raw data and the source experiment's NEW body
- ... On PASS, the `/issue` skill sets `status='awaiting_promotion'` ...
+ Do NOT change status yourself. Leave the task at `interpreting`. After you
+ return, the /issue orchestrator runs interpretation-critic -> clean-result-
+ critic (the final gate); only the orchestrator advances to awaiting_promotion.
+ You MUST NOT run task.py promote — awaiting_promotion is user-only.

    Confidence: high

  5. Target: .claude/agents/experimenter.md — what: add a resumed-pod recovery checklist (the day's #475/#463 infra storm), promoting four uncommitted agent-memory notes into the agent file. Why: Session b77d199a [H38] "the resumed pod had lost uv (/root/.local/bin is on the ephemeral container FS, wiped by stop/resume)"; 81ce8532 [H28] torch 2.11.0+cu130 vs driver CUDA-12.8 cap → cuda.is_available() False after a stale brief claimed "CUDA True"; 5b78eb5c tool errors rsync: command not found → a result-pull silently returned 0 files; 81ce8532/5b78eb5c repeated Command timeout after 30000ms on ordinary git/ps. Memory files feedback_resumed_pod_uv_wiped.md, feedback_torch_cu130_driver_cu128_mismatch.md exist but aren't in the agent file. Proposed edit:

    + ## Resumed-pod / pre-launch recovery (do BEFORE any uv run on the pod)
+ 1. uv survives? `ssh <pod> 'command -v uv || curl -LsSf https://astral.sh/uv/install.sh | sh'`
+    — stop/resume wipes /root/.local/bin (container overlay); /workspace survives.
+ 2. CUDA compat gate after any uv sync / venv re-pin:
+    `uv run python -c "import torch; assert torch.cuda.is_available(), torch.version.cuda"`.
+    "driver too old" => torch CUDA build exceeds host driver; re-pin to a matching
+    cuXXX wheel. Never trust a prior brief's "CUDA True" line — it goes stale on resume.
+ 3. rsync is NOT on RunPod pytorch images: a missing rsync exits 12 and pulls 0 files
+    silently. Use ssh_download / scp / `tar | ssh`; never read "0 files" as "no results".
+ 4. SSH-MCP ssh_execute defaults to 30s; under training load / slow MooseFS a plain
+    git/ps can exceed it. On a 30000ms timeout, retry once with a longer timeout (or
+    Bash `timeout 120 ssh ...`) before concluding the pod is hung.

    Confidence: high

  6. Target: .claude/agents/experiment-implementer.md — what: a porting checklist for when a child reuses a parent experiment's dispatcher/eval scripts. Why: Three distinct round-N bugs from porting parents: 81ce8532 [H10] N_QUESTIONS_TOTAL_FULL = 3300 < 6000 floor — the smoke 5-row path bypassed the production constant, so a production crash hid behind a smoke PASS; 53909073 [H25] smoke and full phases shared a default --out-dir, so a 3-probe smoke gen-cache silently corrupted the full run (#466 r3); #472 [H46] the +gpu_id CVD clobber piled parallel cells onto GPU 0. Proposed edit:

    + ## Porting checklist (reusing a parent's dispatcher/eval scripts)
+ - Assert every plan-specified floor against the PRODUCTION default code path,
+   not the --smoke path. A smoke PASS does not prove the production default runs.
+ - Smoke and full runs MUST write to distinct out-dirs (smoke/ vs full/). A
+   shared default --out-dir lets a tiny smoke gen-cache corrupt the full run.
+ - Parallel cells: pass +gpu_id=N per process; train/sft.py SETS CVD from
+   cfg.gpu_id (default 0), so env CUDA_VISIBLE_DEVICES alone piles all cells on GPU 0.

    Confidence: medium

  7. Target: .claude/rules/marker-leakage-measurement.md — what: two porting/measurement traps. Why: (a) Session b77d199a [H34] crash root cause: an eval_one_cell.build_full_ids count(marker)==1 assertion was forked verbatim from #448, where R came from the marker-FREE base model — but #472 reads the DV on the trained model's own on-policy R, which legitimately contains the marker more than once. (b) Session 8ebf9070 [H4] "for the leaks do they have high cos sim/KL divergence?" + [H5] "Make sure KL/JS divergence wasn't computed and run it as a followup" — #405 reported leakage without the persona-distance metric the project's own #404/#458 line makes load-bearing. Proposed edit:

    + When porting a marker rig: a `count(marker)==1` assertion in build_full_ids is
+ valid ONLY when R comes from the marker-FREE base model. If the DV is read on
+ the TRAINED model's own on-policy R (the default for trained-anchor leakage),
+ R can already contain the marker — assert `count>=1`, not `==1`, or the cell crashes.
+ Any clean-result reporting marker/behavior leakage (or its absence) to bystanders
+ MUST also report the JS/KL divergence + cosine similarity of those personas to the
+ source (persona-distance-metrics.md). "No leakage" without the distance distribution
+ is incomplete — state whether the non-leaking personas were near or far.

    Confidence: medium

  8. Target: .claude/skills/issue/SKILL.md — what: (a) Step 8 is NOT skippable on hand-orchestrated runs; (b) a named measurement-gate-bypass protocol. Why: #444 was almost entirely hand-orchestrated (manual SSH relaunches H7/H10/H15/H20/H22), so the normal Step-8 upload-verifier never fired (→ lost datasets, item 1). Separately, Thomas had to manually order gate bypasses 3× — ea1897f8 [H14] "Just remove this gate and continue", [H39] "Just bypass this gate entirely", then [H64] "What were the bypassed measurement gates?" (he later couldn't recall what he'd bypassed). Proposed edit:

    + Step 8 is NOT skippable even when the run was launched/completed by hand
+ (manual SSH relaunch, gate bypass, mid-flight recovery). Before pod.py
+ terminate, ALWAYS dispatch upload-verifier and require a PASS marker.
+ ### Measurement-gate bypass (user override)
+ On user override of a plan-defined measurement-validity gate: post
+ `epm:gate-bypassed v1` (gate name + one-line rationale) so it's recoverable,
+ and carry the bypassed gate as a scope caveat into the clean-result Motivation.
+ Do NOT silently drop the gate or re-ask on each occurrence in the same run.

    Confidence: medium

  9. Target: .claude/agents/analyzer.md — what: for fact/marker-teaching evals, require the response taxonomy to separate didn't-mention / refused / confabulated-contradictory / stated-taught-value instead of an "other answer" bucket. Why: Session ea1897f8 [H65] "if it doesn't mention the number of benches does that count as a failure?" → [H66] "In the plot it's counted as 'other answer'?" → [H67] "can we do a reanalysis which clearly separates...". The original #444 analysis collapsed distinct outcomes into one ambiguous bucket, forcing a full 5-way re-judge after the clean-result already passed review. Proposed edit:

    + For fact/attribute-teaching evals the per-response taxonomy MUST distinguish at
+ minimum: (1) didn't-mention the target, (2) refused, (3) confabulated a
+ contradictory value, (4) stated the taught value. Never collapse (1)-(3) into a
+ single "other"/"failure" bucket — that conflates non-emission with active
+ contradiction and has forced post-hoc re-judges (#444).

    Confidence: medium

  10. Target: .claude/rules/upload-policy.md — what: harden the turner/EDS-dataset $0 provisioning fence so a decrypt failure raises instead of silently firing a paid Claude batch. Why: Session c7f3d370 [H46] "the turner step triggered a Claude batch despite my safeguards" → [H47] Thomas had to hand-fetch the EDS password. This is the second time (the feedback_turner_prep_paid_fallback.md memory is from incident #468 itself) that the EDS decrypt-failure path auto-fired paid regen — --force-step is a floor, not an exclusive selector, so it doesn't disable the paid fallback. Proposed edit:

    + ## Turner/EDS $0 fence (hard)
+ Before ANY turner/medical dataset provision, assert TURNER_EDS_PASSWORD is
+ present AND set the no-paid-regen flag so a decrypt failure RAISES instead of
+ silently firing a paid Claude batch. --force-step is a floor, not an exclusive
+ selector — it does NOT disable the paid fallback (incidents #468, #468-rerun).

    Confidence: medium

  11. Target: .claude/rules/contrastive-negatives.md — what: add a co-resident multi-marker training caveat. Why: Session 0a559ccd [H88]–[H106]: the #464 rig trained two distinct markers into two personas co-resident in one LoRA — a regime the rule doesn't address, and it visibly confused the localization read ("I don't understand why the co-resident thing would make such a difference"). Marker-to-marker interference and shared-LoRA capacity are uncontrolled. Proposed edit:

    + ## Co-resident multi-marker training (caveat)
+ Training >1 distinct marker into >1 distinct persona in a SINGLE adapter
+ (e.g. ※→pirate, ¶→villain co-resident, #464) is a different regime than the
+ 1-marker recipes above. Marker-to-marker interference and shared-LoRA capacity
+ are uncontrolled; localization measured this way is NOT directly comparable to
+ the single-marker contrastive results. Flag as a scope caveat in the clean-result.

    Confidence: medium

  12. Target: .claude/agents/codex-code-reviewer.md — what: stop the recurring false "pods.conf reverted" finding at the source. Why: Session 9b401811 [H21]: the reconciler had to DISCARD a Codex "Critical (pods.conf reverted)" after independently verifying git log main..issue-471 -- scripts/pods.conf showed zero commits. pods.conf / pods_ephemeral.json carry session-local pod IP/port churn (they're modified in the live working tree right now). A reconciler memory feedback_codex_stale_branch_artifacts.md exists; the guidance belongs in the Codex prompt. Proposed edit:

    + Do NOT flag scripts/pods.conf or scripts/pods_ephemeral.json as "reverted" /
+ "deleted" on an issue branch — they carry session-local pod IP/port churn that
+ diverges from main constantly. Verify with `git log main..issue-<N> -- <file>`
+ (zero commits = not touched by this branch) before raising any such finding.

    Confidence: medium

  13. Target: .claude/skills/clean-results/SPEC.md (voice section) — what: extend the existing byte identical ban to agent verdict/marker prose, not only clean-result bodies. Why: The phrase reappeared in code-review/verdict markers across c7f3d370 [H11] and ae9d3712 [H11]. It's banned in bodies; the risk is it propagates from a review marker into a promoted body. Low stakes (it stayed in markers, which is allowed), so this is the lowest-priority item. Proposed edit:

    + Note: "byte identical" / "byte-identical" is banned in clean-result bodies and
+ should be avoided in verdict/marker prose too — say "matches the #N snapshot" or
+ give the sha256. Do not let the phrase propagate from a review marker into a body.

    Confidence: low

Other problems & notes

  • Agent type 'independent-reviewer' not found (fba6d4cc tool error). One-off mis-spawn: the orchestrator tried Agent(subagent_type="independent-reviewer"), but independent-reviewer is a skill, not an agent (correctly listed in agent frontmatter skills: lists). Action: trace which orchestrator/skill call spawned it as an agent and correct to a general-purpose reviewer that loads the skill; the fix likely lives in a global skill (~/.claude/skills/self-review), which is out of /daily scope → route to /memory-sleep.
  • Dashboard sign-in doesn't honor ?next= (45eda23f [H9]–[H14]): login works but doesn't redirect to the requested page. Action: fix the next= redirect in the Next.js dashboard/ auth handler (out of workflow scope).
  • task.py view --json / REGISTRY crash on #469 (cd3bdec0: TypeError: 'NoneType' object is not subscriptable with has_clean_result: None; KeyError: '469'). Action: task-state, not workflow-file — run task.py audit + re-sync #469's REGISTRY row (feedback_registry_denormalization.md pattern).
  • Codex twin dispatch flakiness — exit 4/8/1 across 0a559ccd, c7f3d370, b77d199a, 9b401811 (Codex code-reviewer / clean-result-critic / methodology-critic). Claude reviewers carried every round so nothing blocked. Action: transient; if it persists, check codex_task.py / Codex-plugin auth health — and if Codex is down, log once and proceed Claude-only rather than retrying into the same failure.
  • #444 hand-orchestrated run was the root of both the lost datasets and the context exhaustion (~96 turns, many manual relaunches). Action: the item-1 + item-8 fixes target this root cause.
  • #475 infra storm (81ce8532): MooseFS errno 116 on uv sync, torch cu130 vs driver cu128, SUPPLY_CONSTRAINT re-provisions. Action: deps fix (cu129 re-pin, transformers>=5.6) landed on issue-475; watch that #475 lands on a driver-580 host or stays pinned to cu129. Memory files saved.
  • #467 strong-NL author length-band unachievable (935a6bfc [H52]): ±20% band + FAIL_LENGTH gate were unachievable against Sonnet's soft length control; caused SMOKE FAILs across rounds 7-8. Action: experiment-code, fixed; feedback_sonnet_strong_nl_length_band_too_tight.md saved.
  • Recurring silent-None-fallback bugs in fresh experiment scripts (#470 ae9d3712 [H11]/[H14], #465 360351d6 [H13]): smoke artifacts poisoning production, row[...] = None on missing keys — violate the project's fail-fast rule. Caught by the ensemble each time. Action: experiment-code class; the item-6 porting checklist + fail-fast discipline cover it, but new dispatchers keep re-introducing it.
  • #463 predictor headline flip-flopped v1→v2→v3 (5b78eb5c): reversed to significant, then back to LOW after the interp-critic found a content-leakage alternative + a raw-vs-partial-rho discrepancy. Action: none — the interp-critic loop worked; promoted at LOW with the caveat binding.
  • #471 anchor is saturated even on the contrastive-negative personas (9b401811 [H32]/[H33]): the #448 saturation trap. Action: before any selectivity claim, #471 must move to a less-trained anchor or a non-saturating DV (full-vocab KL-from-base) and flag saturation as a hard scope limit.
  • #474 preflight hung silently >2 min on MooseFS git-status (48e6ea72 [H36]): killed and proceeded. Action: consider a hard timeout + heartbeat in orchestrate/preflight.py (becomes a workflow proposal if it recurs).
  • The /daily digest-extraction step ran without uv on PATH (this run, 9270451c: command not found: uv). Action: one-off — prefix bare uv with the full path / export PATH="$HOME/.local/bin:$PATH" in the digest step (same class as the known pod uv-PATH gap).
  • MEMORY.md + CLAUDE.md both tripped the 40k-char warning (b81e7171 [H0]): restructured in-session. Action: monitor that both stay trimmed; one MCP /doctor item remains.

My thoughts

Highlighted results

  • no results promoted today