Daily — 2026-06-17 (4 follow-up loops re-parked,

What happened

• Heavy autonomous fleet day, mostly self-healing. Four same-issue follow-up loops completed and re-parked at awaiting_promotion: #642 (onpolicy-matchedlr-rank-isolation — interp + clean-result ensembles both PASS), #545 (metric-race, after a 6-OOM grind that pivoted to a subprocess-isolated vLLM worker — the genuinely-correct fix), #654 (length-matched-dummy-query-control, a 0.5-GPU-h cheap-band auto-run folded into the parent), #653 (rank-ladder read/write — clean-result critique PASS, methodology doc generated).
• #656 (infra) filed + auto-completed per Thomas's PM-session directive: the GCP auto-lane now walks a cost-ordered fallback ladder (on-demand A100-80 → on-demand A100-40 → spot A100-80 → spot A100-40 → SLURM → RunPod terminal rung), reversing the old no-auto-RunPod invariant. Landed in CLAUDE.md + backends/gcp.py + router tests.
• #653 evening: Thomas manually kicked off a new same-issue follow-up loop (install-validated-reladder) to fix the parent's 4 known defects (dose-to-target install with a per-cell floor, restored full-FT rung, upload per-cell install completions, ≥2 seeds on headline cells).
• #651 blocked, needs_user: 5th HF Hub 504 gateway timeout after the dispatcher recovery cap exhausted (~13 h of intermittent Hub failures during multi-arm activation extraction).
• In-flight workflow hardening (already merged today, not by this run): verify_task_body.py checks 22/23 (#537, figure-SHA-vs-Reproducibility-commit consistency + HF revision-pin existence); multiple #653 GCP-salvage fixes (setsid+sudo PATH loss, pgrep self-match, subprocess decode, branch-pinned SLURM guard) auto-applied via the workflow-fix-on-bug protocol.

Applied workflow improvements

1. Target: .claude/skills/issue/SKILL.md — what: harden the Step 9a-quater methodology-doc no-secrets pre-scan to $REPO_ROOT-absolute paths — commit: 04769f767b Why: #654 session a9e2d80e 05:19 — the documented bare scripts/check_no_secret_shaped_strings.py invocation, run from a worktree (or after a removed-dir getcwd miss), resolved to /scripts/check_no_secret_shaped_strings.py: [Errno 2] No such file or directory (exit 2). Self-recovered 6s later by re-running with $REPO_ROOT, but the doc itself taught the fragile form. Matches the project's standing absolute-path discipline for scripts/... invocations. Applied edit:

      `.env` content — with the canonical scanner:
   -  `uv run python scripts/check_no_secret_shaped_strings.py docs/methodology/issue_<N>.md`
   -  (exit 0 = clean, exit 1 = hit). Do NOT use `redact_for_gist.py` for
   +  `uv run python "$REPO_ROOT/scripts/check_no_secret_shaped_strings.py" "$REPO_ROOT/docs/methodology/issue_<N>.md"`
   +  (exit 0 = clean, exit 1 = hit). Use `$REPO_ROOT`-absolute paths — a
   +  bare `scripts/...` resolves against the orchestrator's cwd, which on a
   +  worktree (or after a removed-dir `getcwd` miss) is NOT repo root and
   +  resolves to `/scripts/...: No such file or directory` (incident #654,
   +  2026-06-17, self-recovered after a retry). Do NOT use `redact_for_gist.py` for
   

   Lint gate (workflow_lint.py --check-references + --check-asks): both PASS.

Other problems & notes

• Auto-followup expectation mismatch (#653, session 33364133). Thomas: "run a followup to fix all the problems with this experiment, actually i feel like an automatic followup shouldve run." Diagnosis was correct: the free 0-GPU follow-up (epm:free-analysis-followup-run, Step 9a-ter) did fire, but the GPU-backed follow-up machinery (follow-up-proposer + cheap-band auto-run) lives at Step 10, which only runs after the user promotes — and #653 is parked at awaiting_promotion waiting on Thomas. Held — judgment call: whether fix-up follow-ups should auto-propose/run before the promotion gate is a deliberate-gate-semantics / high-blast-radius design question, not an obvious bug. Suggested action: decide with Thomas whether the proposer should run at the awaiting_promotion park point for flawed experiments (so cheap same-question fix-ups auto-fire pre-promotion), or whether promotion-first is intended.
• Research-idea misroute (#655, PM session 07948087). A new exploratory idea ("Do persona/trait SAE features persist across token positions?") was filed as an EPS [exp] task; Thomas: "No I want you to add it as a research idea in the external research idea store" (~/personal-assistant/research-ideas). It was added there correctly after the correction. Held — judgment call: the boundary between external research-idea store vs EPS proposed task vs docs/research_ideas.md isn't crisply defined in the routing rules; guessing a boundary risks misrouting future captures. Suggested action: Thomas defines the routing rule, then it goes into research-pm.md / the "Routing experiment intent" section. Also: #655 still lingers as a kind: experiment task in the EPS tree — Thomas wanted it only in the external store; consider task.py set-status 655 archived (held: status mutation needs explicit ask per /daily rules).
• No task.py set-backend subcommand (PM session 07948087). Flipping #654 to RunPod required hand-editing backend: frontmatter ("No set-backend subcommand and #654's backend is null"). Held — judgment call: adding a task.py subcommand is a public-API/contract change → architectural, needs greenlight. Suggested action: if backend-flipping recurs, add task.py set-backend <N> <lane> (+ a workflow event), with Thomas's OK.
• gcloud logging read dead-end recurred despite being documented (#642, sessions e4d93701/870d603c, ~4+ retries × 2 sessions). Orchestrator tried gcloud logging read 'resource.type="gce_instance"...' to inspect a GCE instance and got PERMISSION_DENIED: Permission denied for all log views (eps-router@ SA has no log-view IAM); recovered via SSH tail. The lesson IS already in gotchas.md L52 (buried in a #640 guestTerminate entry) — so this is a discoverability gap, not a missing rule. Suggested action (low priority): lift it into a short standalone greppable line if it recurs again; not worth a duplicate now.
• HF Hub extract-dispatcher has no 429/504 rate-limit backoff (#651). Activation extraction over 40+ adapter cells hammered the Hub API → 429 Too Many Requests (2500/5min) killed the dispatcher pid (recovered via warm-cache restart), then a 504 5th-failure block at 22:12. Held — couldn't verify tonight: this is experiment-specific code (scripts/issue_651/...) on a currently-blocked task with no pod/data to smoke against. Suggested action: add HF-Hub rate-limit backoff to the extraction fetch path when #651 is unblocked (it's the same Hub-throttle class likely to recur on any multi-arm activation-extraction run).
• SLURM lane can't serve branch-pinned runs (#642). Branch-pinned dispatchers need /workspace; all three free SLURM lanes (nibi/fir/mila) refuse a non-main repo_branch, so a GCP stockout cascades straight to no_compute_available with no free-lane backstop. Held — architectural: the guard is already in (gotchas.md L55, SlurmBackend._assert_repo_branch_synced); the full fix (rsync from a feature-branch worktree honoring --repo-branch on SLURM) is a tracked router-design infra task, not a same-turn edit.

My thoughts

Highlighted results

• no results promoted today — four follow-up loops (#642, #545, #653, #654) completed and re-parked at awaiting_promotion; promotion is the manual user gate.