EPS
← All docs·Activity

Daily — 2026-07-07 (27 workflow fixes merged; Codex quota outage; 2 refusal wedges; 4 science rounds landed)

updated: 2026-07-08

What happened

  • Science rounds landed on 4 issues. #928 folded the matched-length answer-span control (headline retitled: conditioning on realized CoT predicts the thinking model's answer state no better than a matched-length slice of the answer's own opening; deficit negative in all 50 contexts, 1.4 GPU-h realized vs 3 budgeted). #825 ran two separator-control follow-up rounds (base + on-policy; D_base = 0.590 / D_inst = 0.428 partial provenance effect) and armed a temperature-sampled robustness round on Thomas's directive. #1090 ran three follow-up rounds (margin companion + qwen organism, dose extension, posonly×contexts matrix) and re-parked at awaiting_promotion. #1074 folded its install-dose-extension round (4,875 batch judge calls) → awaiting_promotion.
  • #1112 (LoRA vs full-FT activation-shift geometry) filed, re-scoped by Thomas to a 2×2 on #1090's data, and run through a 7-round crash-fix cascade (degenerate-spectrum smoke crash, CVD-clobber full-FT OOM, ENOSPC at checkpoint rung 24/30 on a $16/hr pod, tokenizer TypeError, single-crasher-masked capture death) — upload-verification PASSed, pod terminated, now at verifying/interpreting. #1092 built its 7.6k-line real-corpus pipeline through 8 review rounds + a 4-attempt WildChat/LMSYS corpus build; GPU launch waiting on fleet-wide A100/H100 capacity.
  • 27 infra/workflow-fix tasks completed + merged in one day (#1096–#1124 band): refusal-wedge watcher lane (#1104), sonnet model-switch refusal rung (#1106), HF 100k-file overflow fallback (#1108), ghost follow-up-label parse fixes (#1109/#1111/#1120), H1-title sync check (#1110), width-aware GCP auto-router with 8×A100 rungs (#1121, from Thomas's "almost infinite credits" directive), GCP handle-sidecar + queue-vanish failover fixes (#1122/#1116), stream-checkpointing gate (#1119), 3 gotchas entries (#1113/#1123/#1124), and more.
  • OpenAI Codex org quota exhausted ~17:00Z until 2026-08-06 — every Codex review twin is a no-show fleet-wide; all ensemble sites run the documented Claude-only fallback. Filed as #1126 (sentinel) + #1140 (account decision, needs you).
  • Two orchestrator refusal-wedges ("violative cyber content"): #1098 dead 08:33–11:13Z, #1090 dead 16:31–19:06Z (~6h wall combined), plus ~12 subagent refusal kills; mitigations #1104/#1106/#1115 merged same-day, residual filed as #1127.
  • HF canonical model repo hit the 100,000-file hard limit; #1090's session rescued 2.6 GB to a private overflow repo within the grace window and #1108 shipped the fallback same-day. Canonical-repo cleanup decision held for you (#1141).

Applied workflow improvements

  1. Target: .claude/skills/issue/SKILL.md — what: Step 9a-quater now says up front to compose body-pointer edits in a staged copy and apply ONLY via task.py set-body, never a raw body.md write — commit: 0cdba98aaa Why: the #1090 session (c86ff35c, 06:59Z) inserted the top-of-body **Methodology:** pointer with a raw pathlib write, self-caught the bypass, and its revert attempt was hook-blocked (~3 turns lost); the mechanism was only named at the END of the step. Applied edit:

    +   Compose both edits in a staged copy (`/tmp/...`) and apply ONLY via
+   `task.py set-body <N> --file ...` (named again below) — NEVER a raw
+   `body.md` write (incident #1090, 2026-07-07: a direct pathlib write
+   bypassed task.py and the revert attempt was hook-blocked; recovery
+   cost ~3 turns).

    Verification: doc-only; workflow_lint.py --check-references + --check-asks PASS repo-wide after commit.

  2. Target: CLAUDE.md — what: the spurious-refusals bullet now names the second observed trigger class — shell-attack/guard-task vocabulary refused as "violative cyber content" — so first-pass briefs for guard-script tasks thin that wording — commit: b37df90147 Why: #1098 (4c1c98c3): 2 reviewer kills then a ~2.7h orchestrator wedge, all triggered by the guard hook's own destructive-git fixture text; the bullet previously named only safety-research vocabulary. Applied edit:

    - ...safety-research vocabulary (backdoor / trigger / implant / misalignment, raw EM completions) sometimes trips...
+ ...safety-research vocabulary (backdoor / trigger / implant / misalignment, raw EM completions) — and, on workflow-guard/security tasks, shell-attack vocabulary (write-then-execute channels, destructive-git payloads, fail-open redirects; refused as "violative cyber content", #1098 2026-07-07: 2 reviewer kills then a ~2.7h orchestrator wedge) — sometimes trips...

    Verification: doc-only; lint gates PASS.

  3. filed for review #1126: Codex quota-exhausted sentinel short-circuit in codex_task.py — 8+ sessions each re-discovered the dead backend by dispatching doomed jobs (dispatched: session spawned).

  4. filed for review #1127: refusal-wedge watcher lane missed both live wedges today (#1098, #1090) — replay the real transcript tails through decide_prompt_wedge, stop dying turns' heartbeats resetting the staleness clock (dispatch pending — watcher backstop).

  5. filed for review #1128: extend guard_repo_root_branch.sh to block repo-root branch merges (conflict markers sat in the shared tree ~70s in #1090) (dispatch pending).

  6. filed for review #1129: raise/derive the Step 9c pristine-oracle timeout default (600s vs ~13min actual for test_workflow_lint.py, #1098) (dispatch pending).

  7. filed for review #1130: codex_task.py forwards post-marker child success-stderr (routes the recursion-guard-parked #1100 candidate) (dispatch pending).

  8. filed for review #1131: /daily route-2 sweep checks for retraction markers before filing (routes the parked #1101 candidate; #1101 was filed from a premise retracted 37s after mining) (dispatch pending).

  9. filed for review #1132: a /daily pass that sweeps recursion-guard-parked workflow-fix candidates (two sat unrouted until tonight's manual mining) (dispatch pending).

  10. filed for review #1133: dose-ladder checkpoint-retention clause in plan-compute-sizing.md (#1112 kept all 30 full-FT rungs → ENOSPC) (dispatch pending).

  11. filed for review #1134: codify or ban the "adopt the more severe verdict without a reconciler" shortcut the #825 session used (dispatched: session spawned).

  12. filed for review #1135: gotchas entry for the CVD-clobber-at-library-train-seam class (14/34 cells OOMed on one GPU in #1090 launch 3; #1112 full-FT saw 1 GPU) (dispatch pending).

  13. filed for review #1136: set-body warns when a kind: experiment body update drops the ## Goal H2 (#1112 re-scope) (dispatch pending).

  14. filed for review #1137: suppress watcher ALIVE-BUT-STALLED alerts when the task's latest non-watcher marker is fresh (3 noise alerts on #1092 today) (dispatch pending).

  15. filed for review #1138: verbatim bare push/merge snippets in the /issue SKILL.md templates (~10 piped-git hook blocks across ≥8 sessions today) (dispatch pending).

  16. filed for review #1139: ad-hoc result summaries state per-arm data/completion provenance (Thomas asked "on-policy?" twice on #825; "single stochastic sample?" on #779 figures) (dispatch pending).

Other problems & notes

  • Route 3 — held for you, filed #1140 (spends money): OpenAI Codex quota exhausted until 2026-08-06 6:26 AM; all 5 doubled review sites run Claude-only until then. Decide: top up / upgrade the OpenAI plan, or explicitly accept single-family review for the month. Cross-family diversity is the project's stated strongest oversight asset.
  • Route 3 — held for you, filed #1141 (destructive/irreversible): the canonical HF model repo superkaiba1/explore-persona-space is frozen at the 100k-file limit; new adapters land in the private overflow repo (#1108 fallback). Decide purge/archive strategy (the parked adapters/issue_397 242 GB review is one candidate).
  • Refusal pressure was the day's dominant tax: ~12 subagent refusal kills + 2 orchestrator wedges (~6h combined wall on #1098/#1090) + a refusal storm that ate ~every other wake on #1090's overnight session. Same-day mitigations landed (#1104 wedge lane, #1106 sonnet-pin rung, #1115 read-hygiene, plus tonight's route-1 CLAUDE.md trigger-class addition); residual predicate fix is #1127. Watch whether tomorrow's kill count drops.
  • Near-miss record (fixed same-day): Thomas's directed #1090 expansion was silently dropped when the task re-parked — semicolon-inline followup-scope notes parsed no followup_label (ghost pseudo-labels; #825 had the sibling literal-\n bug that nearly re-dispatched a finished 4-GPU-h round). Both parse bugs fixed (#1111, #1120); PM caught and re-armed the expansion.
  • Compute waste, bounded: #1090 launch 3 burned ~20 min of 8×H100 on the CVD co-location bug (gotchas filing #1135); a GCP FLEX_START instance vanished server-side while PENDING (failover trigger shipped, #1116); #825's first launch lost ~1.1 spot-A100-h to preemption (accepted spot economics); #1090's fu3 GCP attempt sat 40 min past the queue fence because a sidecar rewrite dropped workload_cmd (fixed, #1122); an idle-GPU advisory fired on #825's >54-min CPU fit phase riding a 1×A100 instance.
  • #1092 parked blocked at the review cap-5 with a reconciler-upheld residual + written fix recipe; Thomas greenlit the fix and respawned it; evening GPU launch hit fleet-wide capacity exhaustion (GCP + SLURM + RunPod all dry; known #653 nibi feature-branch limitation), now queued on an 8×H100 FLEX_START.
  • PM scope churn on #1112: the initial dispatch guessed #642/#606 data where Thomas meant #1090's; ~25 min churn, no GPU wasted. Pattern note: when a NEW-direction capture stems from a discussion anchored on a specific issue's data, confirm that issue's data as the substrate before dispatching.
  • Interactive pushback recovered well: on the #779 mentor-reply draft, my anisotropy critique overstated; Thomas pushed back, I conceded and derived the ≈0.85–0.90 estimate from the existing per-rank curve. One wasted round-trip on a Dan-bound draft.
  • Watch items (note-only): 26 no-op true Bash calls in the #1112 session (wasted round-trips; watch for recurrence before touching skill text); a VM-side detached launch captured the wrong PID (bash -c+setsid parent) and reported a false failure on a succeeded phase — the #813 pid-file contract applies to ad-hoc VM detaches too; TaskOutput(task_id="placeholder") slip in two sessions; several gh pr create-before-push slips (all self-healed ≤2 turns); #825's GCE workload soft-failed its git push (no credentials) and wrote a phantom final_commit_sha — lives in experiment-side dispatch scripts, upload-verifier caught the artifact gap.
  • Guards earned their keep today: piped-git hook 10/10 blocks converted to bare re-runs (filing #1138 targets the habit); Step 10d Guard 3 caught 57 stale pre-rebase duplicate commits on issue-1105; the branch guard blocked a working-tree revert on the shared root; upload-verifier caught pod-only artifacts on #825, #1090 and #1112 before teardown.

Living-docs drift

  • living_docs.py check FAILs with 199 findings (drift hash 078285c14444): ~75 dead legacy anchors (relates_to pointing at removed a1/b1/c1-style questions), ~54 completed clean-results absent from any question's evidence, dozens of relates_to ⇄ evidence mismatches (incl. #722/#763/#494–#501), and the stale spec-role-header State trailer (2026-06-02 vs results promoted through 2026-06-16).
  • Materially the same drift as the still-open re-synthesis proposals logged 2026-06-29 (1736fca0804f), 2026-07-04 (d2a80db46b3b), and 2026-07-06 (b009cef9ecd9) — the hash moved only because task titles/counts shifted. Not re-proposing separately; the open proposal stands: re-synthesize open_questions.md (living-docs updater/backfill) — needs your ok; not auto-applied.
  • No parked living-docs rejections or parked-redundant follow-ups became re-proposal candidates today.

My thoughts

Highlighted results

  • no results promoted today