EPS
← All tasks·#454Completed

Migrate clean-result body to the 2-content-section spec (TL;DR + Reproducibility) — #432 overhaul [weekly-W22 #1]

kind: infra
track:

Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.

No comments yet.

Source: 2026-W22 weekly proposal #1. Thomas dictated a wholesale clean-result body overhaul against task #432: "no separate Details section; parameters into Reproducibility; Methodology corrections not necessary; just # Human TLDR (empty) + # TLDR (LessWrong style, starts with Motivation, per-result subheadlines with one plot each, one raw-completion example per artifact + dropdown + links to all raw); Next steps not by default; remove 'why confidence is where it is'; training/eval data as a 5-example dropdown + link in the relevant section; every clean result except Motivation stands alone; never use 'byte identical'."

This must ship as ONE reviewed+tested change: the canonical verifier scripts/verify_task_body.py enforces the CURRENT 4-section model, so changing the docs without the verifier (or vice versa) breaks the promotion pipeline.

The full implementation plan (check-by-check verifier changes, doc edits, test rewrites, migration analysis, execution order) follows.


Plan: Migrate clean-result body to the 2-section spec (TL;DR + Reproducibility)

0. Executive summary of what I found (decision-critical)

  • The verifier is a forward-only gate. It runs ONLY at /issue Step 5 (analyzer pre-publish) and clean-result-critic pre-pass. It is NEVER run retroactively over the ~95 existing has_clean_result: true bodies. I confirmed those legacy bodies already FAIL the current verifier (they carry retired shapes: ## AI TL;DR, ## Source issues, ## Headline numbers, legacy-sagan-card HTML). So no migration / grandfathering work is needed — changing the verifier cannot break already-promoted bodies. This is the load-bearing finding that de-risks the whole change.
  • Task #432 is the target END state, not a currently-passing body. I ran it through the current verifier: it FAILs 5/19 checks (missing ## Details, no Motivation/What I ran/Results bullet labels, etc.). #432 already embodies most of what Thomas dictated (no Details, params in Reproducibility, findings as #### subheadlines with one plot each, no "why confidence", no Next steps). The new verifier must make #432-shaped bodies PASS.
  • #432 still has two stale pieces the new spec must also fix: (a) the old ## Human TL;DR stub uses Headline/Takeaways/How-this-updates-me — the dictation says Human TL;DR is empty; (b) the new spec wants TL;DR to start with a ### Motivation (or **Motivation:**) labeled block then findings.
  • "byte identical" appears in ~10 legacy bodies as a data-equivalence phrase. Thomas wants it banned going forward -> a new audit_clean_results_body_discipline.py PATTERN, not retroactive edits.

1. The exact new required body shape

---
<frontmatter: title, kind, goal, has_clean_result, ...>
---
# <one-sentence claim> (LOW|MODERATE|HIGH confidence)

## Human TL;DR

placeholder            <- single stub token; Thomas fills in (UNCHANGED behavior)

## TL;DR

### Motivation          <- REQUIRED, labeled literally "Motivation", comes FIRST
<LessWrong-style narrative: why this matters, prior tasks linked>

### <result-1 subheadline naming the finding>
<setup prose>
![alt](permanent-url)            <- exactly one plot per result
> **Figure.** *italic lead.* caption...
<read prose>
<one cherry-picked raw-completion example + dropdown of more + link to all raw>

### <result-2 subheadline ...>
<same shape, one plot>
...

## Reproducibility
**Parameters:** <table — ABSORBS the old Details parameters table>
**Artifacts:** <incl. training/eval data dropdown-of-5 + full-link, placed in relevant section>
**Compute:** ...
**Code:** ...

Rules distilled from the dictation:

  • Two H2 sections only: ## Human TL;DR, ## TL;DR, ## Reproducibility. (Three H2s total; "two-section" = the two content sections beyond the human stub. Call it the 2-content-section model.)
  • No ## Details. Everything that was in Details (narrative, figures, samples, "why this test") moves UP into ## TL;DR under per-result ### subheadlines.
  • No ## Figure H2 (already deprecated).
  • Parameters table -> ## Reproducibility (under **Parameters:**, as #432 already does).
  • No "Methodology corrections" section by default. If a correction is load-bearing it folds into the relevant result's prose — no dedicated H3.
  • No "Next steps" by default. Keep the hard exception: raw-completions-not-uploaded still requires the re-run note, but it lives in the relevant result's prose now.
  • No "Why confidence is where it is" section. The Confidence: LEVEL — rationale sentence moves to ## Reproducibility (or the end of the last TL;DR result) — verifier still cross-checks it against the title.
  • TL;DR starts with ### Motivation labeled as Motivation.
  • One plot per result; one cherry-picked raw-completion example per artifact, a dropdown (<details>) of more, and a link to all raw completions.
  • Training/eval data: a <details> dropdown of 5 examples + a full-data link, placed in the relevant result section (#432 already does this — keep).
  • Every clean result except Motivation must stand alone and never use "byte identical" (or "byte-identical").

2. verify_task_body.py — exact check-by-check changes

Reference current check list (CHECKS, 17 body checks + Goal soft check = 19 results). By current name/number:

#Check (current)ActionDetail
0check_body_nonstubKEEPunchanged
0bcheck_no_duplicate_frontmatterKEEPunchanged
1check_title_confidenceKEEPunchanged
2check_required_sectionsCHANGEREQUIRED_H2_SECTIONS = ["Human TL;DR", "TL;DR", "Reproducibility"] (drop "Details"). Order check follows. Remove the ## Figure-between-TL;DR-and-Details positional clause.
3check_tldr_labelsCHANGENew required label set. Drop What I ran, Results as required bullets. New TLDR_BULLETS_REQUIRED = ["Motivation"]. Motivation may be ### Motivation H3 or **Motivation:** bullet — relax the regex to accept an ### Motivation heading too.
4check_figure_imageCHANGEImage must exist inline under ## TL;DR. Keep >=1 image required.
4bcheck_figure_url_resolvableKEEP (retarget)Now scans images under ## TL;DR only.
5check_figure_captionKEEP/SIMPLIFYVacuously passes when ## Figure absent (always now). Keep minimal.
6check_confidence_matchesCHANGECurrently looks for Confidence: line inside ## Details. Retarget to whole-body (or Reproducibility + TL;DR). Keep title-match + >=20-char rationale.
7check_repro_subgroupsKEEPunchanged.
8check_repro_url_permanenceKEEPunchanged.
9check_repro_sentinel_scrubKEEPunchanged.
10check_cherry_picked_labelCHANGERetarget sample-fence scan from ## Details to ## TL;DR.
11check_qualitative_data_linkCHANGESame retarget to ## TL;DR.
11bcheck_planned_vs_actual_denominatorCHANGERetarget section scan from Details to whole-body so it still fires if a corrections paragraph is kept. Document that the H3 is no longer required.
12check_figure_h2_is_deprecatedKEEPStill WARNs on legacy ## Figure.
13check_details_narrative_flowCHANGERetarget figure-dump + bad-H3-label heuristics to scan ## TL;DR.
14check_mdx_safe_urlsKEEPunchanged.
15check_repro_committed_claims_existKEEPNEW as of weekly-W22 #2; unchanged here.
check_goal_presentKEEPunchanged (soft WARN).

Helper changes: section_text(body, "Details") callers (6, 10, 11, 11b, 13) retarget to "TL;DR"/whole-body; _gather_figure_image_urls drop "Figure"; _iter_sample_fences/_prelude_window operate on TL;DR. New constants REQUIRED_H2_SECTIONS, TLDR_BULLETS_REQUIRED. Rewrite module docstring ("Three required H2; Details retired 2026-W22"). Net check count unchanged (retarget, not add/remove) — but note the W22 #2 committed-at-sha check already bumped CHECKS to 18 / results to 20; preserve those counts.

3. Doc edits — SPEC.md, analyzer.md, clean-result-critic.md, codex twin, CLAUDE.md

3a. .claude/skills/clean-results/SPEC.md (source of truth — edit FIRST)

Replace the 4-section enumeration with the 3-H2 / 2-content-section model. Delete the ## Details bullet + sub-rules. Rewrite ## TL;DR to: start with ### Motivation; one ### <finding> per result with one inline figure + one cherry-picked raw example + <details> dropdown + link to all raw. Move Parameters table + Confidence sentence into ## Reproducibility. Rewrite the mechanical-checks list. Rename "Story arc for Details" -> "Story arc for TL;DR results". Add byte_identical to the anti-pattern list. Add a migration note (forward-grandfathered). Make #432 the canonical exemplar.

3b. .claude/agents/analyzer.md

Rewrite "four required H2s" -> three everywhere (~lines 41, 214, 244, 258, 368). Step 6 pre/post-flight greps: change ^## Details$ -> ^## TL;DR$ + ^## Reproducibility$. Delete the "Methodology corrections as last H3 in Details" rule. Confirm Human TL;DR stub stays the bare placeholder. Rewrite TL;DR shape (Motivation first; per-result H3 with one figure + raw example + dropdown + all-raw link; drop What-I-ran/Results required bullets). Move Confidence to Reproducibility. Retitle "Story arc for Details" -> TL;DR result beats. Update anti-patterns #11/#13/#14; add "byte identical" ban.

3c. .claude/agents/clean-result-critic.md

Update mechanical pre-pass section names. Fold Lens 4 (Details narrative) rules into Lens 2 (TL;DR); drop the Confidence-after-Parameters sub-rule. Lens 8: drop the Methodology-corrections-placement half, keep the title check, rename "Mentor-facing title". Lens 12: retarget ## Details -> ## TL;DR result beats. Lens 13: drop the ### Methodology corrections H3 sub-checks; scope-shrinkage documented in result prose. Add a "byte identical" ban (Lens 6 voice). Keep lens numbers stable, retitle/mark-merged to minimize cross-file churn.

3d. .claude/agents/codex-clean-result-critic.md

Mirror every clean-result-critic change (per-lens summaries at ~156, ~172-178, ~199-256, ~273-287). Keep it a thin mirror.

3e. CLAUDE.md

§ Experiment Report Structure: "four required H2 ... / TL;DR / Details / Reproducibility" -> three; point at SPEC.md. § After Every Experiment item 8: missing-condition documentation now in result prose, not ### Methodology corrections.

3f. .claude/workflow.yaml

Update the clean-result-critic marker fields description (lens set + 2-content-section shape) and the stale verification: "...six mechanical checks" line.

4. Test updates under tests/

tests/test_verify_task_body.py: rewrite GOOD_BODY to the new shape (drop ## Goal/## Figure/## Details; TL;DR starts ### Motivation then a ### <finding> with inline image + caption + cherry-picked sample + qualitative-data link + <details>; Parameters + Confidence in Reproducibility). Fix all dependents: section-order/missing-section tests, test_missing_tldr_labels (target Motivation), cherry-picked + qualitative-data injection points (-> TL;DR), confidence tests (locate in Reproducibility), test_goal_* (GOOD_BODY no longer carries ## Goal — inject a copy). Add new tests: ### Motivation form passes; a #432-shaped body passes end-to-end; pin the stray-## Details behavior (recommend FAIL). New audit test: byte_identical fires. Also audit tests/test_verify_clean_result.py, test_verify_clean_result_body_stdin.py, test_clean_result_critic_planned_vs_actual.py (retarget Methodology-in-Details fixtures), test_task_cli_set_body_assertions.py.

5. Migration — none needed

Verifier is forward-only; the ~95 legacy bodies already FAIL it and are never re-verified, so tightening cannot regress them. Keep the legacy-sagan-card short-circuit. Leave "byte identical" in legacy bodies (forward audit-pattern only). One consequence to flag: in-flight awaiting_promotion tasks (54) with a 4-section draft will FAIL on next analyzer/critic re-run and simply get re-drafted under the new spec — acceptable (#432, #382, #237 are examples).

6. Recommended execution path

Ship as one reviewed+tested infra-kind /issue change on a worktree. Edit order (SPEC.md first): SPEC.md -> verify_task_body.py + helpers -> audit script (byte_identical) -> tests -> analyzer.md -> clean-result-critic.md -> codex twin -> CLAUDE.md -> workflow.yaml. Gate on green tests + workflow_lint.py --check-asks + ruff. End-to-end smoke: a #432-shaped body PASSes, a legacy 4-section body FAILs cleanly. Keep lens numbers stable. Code-review ensemble verifies docs<->verifier<->tests consistency.

Decision for Thomas before execution: should a stray/leftover ## Details H2 in a NEW body be a hard FAIL (forces clean migration) or an allowed extra H2? Recommend FAIL.

Target exemplar (END state to make PASS): tasks/awaiting_promotion/432/body.md (with Human TL;DR -> bare placeholder and an explicit ### Motivation label; it already uses ### Motivation).

Activity