Measure the finetuned map M+ on the finetuned model's own generations (parked: duplicates #833)
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Overview
Follow-up proposal 3 from task #823's proposer round (2026-07-03), parked on_hold by the single-pass redundancy screen (Claude + Codex, unanimous). Revivable via task.py set-status <this> proposed.
Goal
Determine whether the finetuned per-context map M+ should be measured on the finetuned model's own generations by regenerating answers from the finetuned checkpoints on the same held-out LMSYS contexts, refitting M+ on those answer profiles, and testing whether its fit and read-out geometry differ from the current pre-finetuned-response measurement.
Proposal spec (verbatim from the proposer)
- Hypothesis: given #823's content-indexing result, M+ on finetuned generations differs only insofar as generated content shifts — high retention unless fine-tuning materially changes answer content.
- Falsification: large R²/geometry divergence → the pre-finetuned-response M+ mis-characterizes the finetuned mapping; the predictor line must switch measurement basis.
- Differs from parent: one variable — answer provenance for the M+ fit; harness/contexts/ridge identical. ~4998 contexts, vLLM regen per finetuned checkpoint (temp 1.0, seed 42), TF 28-layer extraction, phase-4 CPU ridge.
- est_gpu_hours: 12 · cost_class: needs-gpu · headline_affecting: no · question_relation: substantially-different
Value critique
Verdict: redundant (unanimous — Claude follow-up-critic + Codex twin, no reconciler needed).
Duplicates task #833 (interpreting): "Does #722's function-change verdict hold with on-policy answers from the fine-tuned model" — same construct + measurement (refit the finetuned map on answer profiles computed from answers GENERATED by each fine-tuned model instead of teacher-forced pre-FT answers; test whether fit/geometry and the measurement-basis verdict change), with a same-or-weaker manipulation: #833 additionally carries the matched-text decomposition control this proposal lacks, and this proposal's checkpoint set is an unresolved scoping decision ("which checkpoints constitute M+"). #811 (followups_running) is the answer-summary-dependence sibling of the same verdict.
Revival condition: if #833 lands with a rig-scope caveat that leaves the #779/#823 LMSYS monitoring surface genuinely uncovered, revive this to run the LMSYS-surface variant.