EPS
← All docs·Activity

Daily — 2026-07-18 (workflow-fix fleet day; 0 results promoted)

updated: 2026-09-03

What happened

  • Workflow-fix fleet day: 33 kind: infra tasks completed (#1487–#1524 range), landing yesterday's 12 /daily route-2 filings (#1493–#1504) plus the watcher-dispatched siblings through the full planner → critic ensemble → implementer → Claude+Codex code-review → Step 10d pipeline. #1497 was archived won't-fix — its filing premise (needs-human tag "dropped at filing") turned out to be a deliberate user-directed mass remove-tag on 07-17; its session surfaced the artifact-state-evidence candidate routed below.
  • #1489 (fact/instruction/persona context augmentations — all fell under the manipulation-check floor) ran end-to-end TODAY: created, executed, and parked at awaiting_promotion with analyzer round 3 folding in.
  • #1417 (context→answer map robustness across rude/evasive/addressee-free/AI-relay framings) is mid-follow-up round at followups_running — plan v4 verified (n_fail=0), refit-mode smoke PASSed unified-architecture check at 23:35Z.
  • #1482 (context→answer map error is category-structured; non-English contexts) advanced through phase p2→p3 at interpreting; #1481 (matched-recipe contrastive vs positive-only across 4 behaviors) is running, walking the GCP on-demand A100-80 rung at ~23:56Z.
  • Gate parks needing Thomas: #1491 (context→answer map vs model scale, 0.5B–14B) at plan_pending; 8 experiments at awaiting_promotion (incl. #1489, #1426, #1335, #1072, #1434); new proposed #1488, #1515.

Applied workflow improvements

Route-1 (trivial mechanical, self-applied — lint gate --check-references + --check-asks PASS after both commits, pushed):

  1. Target: .claude/rules/gotchas.md — what: count-free sibling cross-reference at the pilot-timing entry ("the two smoke/production-parity entries above" → count-free), so the count can't go stale as entries accrete — commit: e6fd1a82a0 Why: #1512's parked candidate (2026-07-18T15:17Z); phrase verified still present at compose time. Applied edit: 
    - Sibling of the two smoke/production-parity entries above (process width #1315/#1333, gate calibration #1345)
+ Sibling of the smoke/production-parity entries above (process width #1315/#1333, gate calibration #1345)
  2. Target: .claude/skills/issue/SKILL.md (2 aligned spots, one commit) — what: (a) merge-failure shape-0 now names the sanctioned wait form ("wait ~20s via a bounded until-loop or a bg-Bash re-check — NEVER a leading foreground sleep") instead of a bare "wait ~20s" that sessions kept implementing as the harness-blocked sleep N && … shape (3 blocked firings on 07-18: #1417 14:03Z, #1511 16:44Z, +1); (b) the ≤45-min wait-segmentation rule now carries the ToolSearch("select:Monitor,TaskOutput") schema-preload pointer (2 sessions burned a turn on the unloaded-deferred-tool InputValidationError: #1066 00:31Z, #1141 00:30Z) — commit: 1ad233fb0c Why: both align skill text with standing CLAUDE.md rules (no-foreground-sleep; load-deferred-schemas-first) — contradiction/pointer fixes, no new behavior.

Route-2 (behavior/logic changes, filed for independent review via /issue --auto; tags wf-fix + daily-auto-filed unless noted; ledger logs/daily/filings-2026-07-18/filed.jsonl; most spawns deferred to the watcher proposed_infra_sweep backstop under the 5-session cap):

From the Step C parked-candidate sweep (16 sweep rows → 10 filings; every source task carries its epm:workflow-fix-task-filed routed-record):

  1. filed for review #1527: verified-at-filing clause (e) "artifact-state mutation check" — absence-of-tag evidence binds only after the task folder's git history rules out a deliberate later mutation (the #1497 false-premise filing). From #1497's parked block, fp 30137f086efe.
  2. filed for review #1528: guard deny sidecar JSONL (ts + arm + bounded redacted head on every deny) so guard false-positive filings stop needing ad-hoc transcript mining. From #1501's parked block.
  3. filed for review #1529: daily_drive_filings.py forwards the filer's sibling-advisory stderr on the success path (currently truncated to out[-300:]). From #1502's parked block.
  4. filed for review #1530: hard HEAD==main assert at the task_workflow COMMIT site + new_worktree.sh never transits the shared HEAD — four task.py commits from three sessions landed on branch issue-1509 during a ~95s worktree-creation window (commits 9fb6221300/76cbf6e3ce/07581c38ef/8d9484a323, all verified off-main). From #1509's parked block.
  5. filed for review #1531: inline-payload-lint-gate contract test scans ALL anchor occurrences (red on pristine main — a #1500-era earlier mention shadows the canonical 9a-ter block). From #1516's parked block.
  6. filed for review #1532: restore the test_ruff_policy full-ruleset helper pin to green (8 errors on pristine main; C901 ×5 + RUF001/2/3 ×3) — PRIMARY of a 6-sighting group (1498-1/1511-1/1516-prose/1520/1523/1524); distinct scope from on_hold #1486 (cross-referenced).
  7. filed for review #1533: implementer Gate-scope duty emits the raw pin-sweep hit FILE LIST (round-1 report omitted 7 hit files the reviewer had to discharge). From #1494's parked prose, confidence: low filed per the any-confidence directive.
  8. filed for review #1534: eval_results/INDEX.md merge=union in .gitattributes (the recurring append-conflict class, incl. at sync_repo_root's own pull). From #1525's parked prose. (The #1417 session also hit the same conflict pair — tests/sparse_cones.txt is worth the spawned session's consideration as a sibling.)
  9. filed for review #1535: plans enumerate every off-pod phase's file reads + a declared-VM-side-phase slot so upload-verifier r1 stops FAILing by construction (merged: #1526's parked prose + #1426's twice-repeated F4 verifier round-trip).
  10. filed for review #1536 (daily-fix:, wf_fix false — experiment/library code): deterministic fail-fast prompt/completion type-homogeneity assert in the SFT data path before SFTTrainer init (code backstop to the #1508 TRL-UB gotcha). From #1508's parked prose.

From the transcript problem sweep (16 candidates → 11 filings; 3 dropped as already-covered, 1 already-landed, 1 merged — see logs/daily/filings-2026-07-18/mining-report.md):

  1. filed for review #1538: guard_repo_root_branch.sh false-positive class 2 — destructive-op literals inside quoted grep PATTERN arguments (blocked a read-only pytest-log grep at 17:00Z, AFTER #1501's heredoc-FP fix merged 10:04Z).
  2. filed for review #1539: never-run/"nothing live" claims must scan live pods + GCE instances + run-markers on PARKED tasks — a status-folder-only scan asserted "nothing live is generating that cell" while #1345's inline round had generated 2,019 Assistant-named stories that night; Thomas caught it.
  3. filed for review #1540: planned-vs-realized cell reconciliation at EACH lane completion — #1481's sycophancy bare arm was never re-swept until Thomas asked ("bare never banded and was never re-swept") and pushed "run it in parallel now".
  4. filed for review #1541: plan §9 sizes boot disk against END-of-run ACCUMULATED footprint — three #1481 GCP lanes died in one day to disk-headroom-exhaustion:200gb-boot-disk after all 24 training cells' outputs accreted.
  5. filed for review #1542: smoke must exercise every non-smoke-fenced load path (or the fenced branch gets a standalone 1-cell probe) — #1481's impolite lane crashed on an IsADirectoryError sitting behind a cfg.smoke fence no smoke could reach.
  6. filed for review #1543: artifact-reuse (h)(iv) staging probe runs PER reused source-family AND per staged consumer — two #1481 staged-layout crashes in one day (datagen_topup/ 404; stage_hub_prefix reread).
  7. filed for review #1544: workflow_lint check flagging upload_file-inside-a-loop in scripts/issue* drivers — #1481 shipped a ~1,400-commit per-file loop that 429-stormed HF despite the prose anti-pattern.
  8. filed for review #1545: root-guard self-test isolation (temp git repo / lock-serialized) — 13 spurious test_check_git_recipes_root_guard_* failures under a concurrent shared-root git op cost #1506 a 17-min full gate re-run.
  9. filed for review #1546: trigger-dense ingestion digests for ORCHESTRATOR poll/forensics turns + guard-hook briefs pass BLOCKED text by file reference — 7 refusal kills wedged the #1481 orchestrator (session replaced at 18:13Z); #1503's same-day fix covers first-pass briefs only.
  10. filed for review #1547 (daily-fix:, wf_fix false): audit ALL direct HF hf_hub_download/upload_* call sites to route through the transient-retry helpers + fleet-shared commit-budget batching guidance + crash-persist retry budget — 3+ independent 429-kill incidents in one day (#1426, #1335, crash-report download, persist=failed_uploads).
  11. filed for review #1548: surface cap-parked screened free-analysis follow-ups (visible epm:progress "parked by cap" note) — #958's TOP-RANKED 0-GPU follow-up sat unrun until Thomas found and kicked it.

Other problems & notes

Full per-problem record: 89 transcripts mined (129 mtime-candidates, 40 dropped by the in-file-timestamp cross-check), ~39 clean, 95 problem blocks → logs/daily/mining-2026-07-18/chunk-{0..5}.findings.md. Route-3 held + notable notes below; retraction re-check: 16 marker-mined Step C candidates scanned, 0 retraction-skipped; 4 mining candidates killed by compose-time freshness checks (listed under "already covered").

Route-3 held (tracked needs-human tasks the PM surfaces):

  • #1537 (daily-held, from #1517's parked block): enforce body presence on wf-fix filings — held because the refusal half touches the task.py new CLI contract (architectural carve-out; the #1173 WARN half is already landed in file_infra_task.py; a filer-only descope fork is named in the body). Suggested action: pick refuse-vs-warn scope.
  • #1549 (daily-held, new): reap the stranded pre-cutover shared-socket tmux server (~55 GiB RSS) + confirm the killed 01:00Z session's open Leg-3 delta-CI verification — held under the destructive carve-out (killing a live server process needs a human eyeball on tmux ls first). The fleet-kill root cause itself (mygoat 12h recycle on the shared socket) was fixed the same day, and the 00:57Z experiment ask the kill dropped WAS re-captured same-day as #1489 (ran end-to-end — verified, no action needed).
  • Codex quota outage until 2026-08-06 — every doubled review site ran single-Claude all day (flagged by 5/6 miner chunks): already tracked in #1472 (daily-held: review diversity during Codex outage to Aug 6, blocked); freshness note posted on #1472. Thomas's recorded stance in-transcript: "wait it out and change the workflow until then".
  • #1141 (HF model repo 100k-file limit): its session found the premise STALE (repo at 117,050 files, accepting uploads since ~07-08) and parked blocked with a decision package — needs Thomas's call on what survives.
  • #1038 (#681 bind-migration cutover): audit found the 07-04 premise stale (/ resized to 945G) and the documented cutover recipe "unexecutable as documented"; parked blocked with a recommendation — needs Thomas. The CLAUDE.md § Disk-hygiene #681-cutover text is stale pending that decision.
  • #1491 parked plan_pending at 490 GPU-h (> the 100 GPU-h auto-approve cap) — the scale-ladder plan awaits Thomas's approval (normal gate, PM surfaces).

Already covered / already landed (no filing; compose-time freshness catches):

  • Inline-gate coverage residual ("why did the gate miss e050ead51c"): the offending figure-script commit landed 00:33Z; #1500's mechanical inline-lint-gate enforcement merged ~6h later the same day and covers the class — already-landed. (#1514 fixed the symptom at 17:04; ≥4 sessions burned pre-existing-red triage in between.)
  • #1434's ~4.4h dead until-loop wait: the ≤45-min wait-segmentation + long-phase heartbeat duty already rules this out — the wedge was a rule violation, not a rule gap; recovered by another session, task completed.
  • Step 10d mergeability: the poll-until-non-UNKNOWN step already exists (landed via #1210); #1335's 13:44Z spurious "merge conflicts" resolved in ~40s.
  • Chat-figure defaults (absolute axes / plain-English legends / setup disclosure): already ruled (CLAUDE.md ad-hoc provenance + no-opaque-codes memory) — today's ~10-question #958 churn + the #1335 legend relabel are compliance lapses, not rule gaps.
  • Thread-caps red (issue1092 dotenv): fixed by #1514 (606fa93f0d); the two parked candidates citing it recorded already-landed.
  • Guard heredoc false-positive (blocked the 07-17 /daily's own probe at 06:41Z): fixed the same morning by #1501 (merged 10:04Z); the residual quoted-pattern FP class is filed as #1538 above.

Notable self-recovered incidents (machinery worked; no filing):

  • Refusal kills: ~20 across the fleet (7 wedging the #1481 orchestrator → watcher fresh-session replacement; 2× #1426 analyzer fold; #1489 upload-verifier at 43 tool calls; #1483 + #1487 fact-checkers; #1493 planner; #1482 implementer; #1335 orchestrator turn; 2× #1500 tick turns; #1215 code-reviewer post-verdict; #1501 subagent + 2 orchestrator turns). All recovered via the existing ladders (durable-verdict-first, thinned retries, rung-(b2) sonnet pins, watcher wedge lanes); systemic residual filed as #1546.
  • #1489 session died at the context ceiling (2× "Prompt is too long", 08:35Z) — healthy pod run handed to the watcher's #1453 instant-respawn lane; three GCP crash rounds earlier in the run were diagnosed + fixed in-session, with the TRL 0.29.1 mixed-schema UB durably documented via #1508 (merged same day).
  • #1482 burned ~18h / 5 GCE attempts on five distinct code bugs (SAE loader FVE; unretried P2 child download; --seed-partial skipping staging side-effects; bookkeeping-file sweep; P3 OOM) — each round followed crash-fix-rounds.md; failure-lessons posted in-session for the consolidate_lessons cron; run completed 00:39Z.
  • #1417: serial dense-QR null loop caught at smoke (961 s/pair → vectorized in-session, the vectorize-many-cell-fits class); Phase C pilot-gate budget re-grounded 1h→1.5h/lane after a measured refute; both recovered in-session.
  • GCP capacity churn absorbed by design: #1072 flex-start queue-vanish relaunch; #1481 casual-s137 boot-loop → no-compute → later re-drive; A100 stockout delaying #1426's restart ~1.9h; sycophancy lane's two unexplained instance deaths (relaunched; cause never established).
  • #1426's F4 VM-side phase upload-verifier round-trips (twice) — auto-recovered; the structural fix is filed as #1535.
  • 13 spurious root-guard self-test failures in #1506's gate (fail-open under a concurrent git op) — clean re-run; structural fix filed as #1545.
  • #1497 false-premise filing archived after its session proved the "dropped needs-human tag" was a deliberate 2026-07-17 PM remove-tag; the corrective clause-(e) fix is filed as #1527.
  • #1517 landed frontmatter-only (filer passed no body); the spawned session reconstructed the spec in ~8 min; the enforcement question is held as #1537.
  • Two Usage-Policy-adjacent user-visible slips fixed in-session: a 404 figure URL handed over after a rebase changed the SHA (#958, re-verified + reissued; the global CLAUDE.md verify-link rule was strengthened the same day), and a "No response requested." reply after a SIGKILL-resume that forced 3 user nudges (#1092 chat; harness resume semantics — no repo-side fix derivable).
  • Title-sync sweep (Step D) ran clean of NEW drift: 8 rows unchanged (see ## Living-docs drift).

Daily stub push note: the skeleton's first push was rejected non-fast-forward and recovered via sync_repo_root.py (state=synced, push succeeded) — no action needed.

Living-docs drift

  • living_docs.py check FAILs with 199 findings (normalized drift hash 078285c14444): ~75 dead legacy anchors (relates_to pointing at removed question anchors, tasks #28–#411 era), ~54 completed clean-results absent from any question's evidence, relates_to ⇄ evidence mismatches on the newer predictor line (#494–#501, #722, #763), and the spec-role-header State trailer stale vs 7 promoted results.
  • drift proposal already open (logged 2026-07-07, hash 078285c14444 — byte-identical drift; earlier open proposals 2026-06-29 1736fca0804f, 2026-07-04 d2a80db46b3b); not re-proposing.
  • Parked-proposal re-consideration (Step B): no epm:living-docs-update-rejected rows without a later epm:living-docs-updated surfaced as re-proposal candidates tonight; no on_hold parked-redundant follow-up met the changed-context bar.
  • Title-sync drift sweep (Step D): title-sync drift unchanged (8 rows: #432 #458 #654 #658 #664 #665 #763 #1226 — see logs/daily/2026-07-11.md for the verbatim rows).

My thoughts

Highlighted results

  • no results promoted today