The marker-leakage channel structure is panel-specific: on held-out personas, geometry routes the training-induced change, and no channel claim transfers beyond the base end-of-answer level persisting through training (MODERATE confidence)
Highlight text on any card (body, plan, or an event) to anchor a comment, or leave a whole-task comment here. Mention @claude to summon a reply.
No comments yet.
Human TL;DR
Headline. the leakage channel structure i found on the 16×26 context panel is real — it survived proper error bars almost number-for-number — but it doesn't travel: on the held-out persona panel, persona geometry predicts the training-induced change in marker pressure, not some pre-existing base-model affinity map, and the one claim that looked like it transferred (the end-of-answer clamp tracking the base margin) dissolved under a partial check into the base end-of-answer level persisting through training.
Takeaways.
- almost every inline number reproduced exactly (the one exception: the pooled cross-validation headline — 0.71 inline, 0.83 on the faithful re-run), but three of the five headline interpretations changed once real error bars were attached — reproducing a number and trusting its story are different things
- the "weak at home, leaks everywhere" story is dead: it was three thin, contentless prompts doing all the work; with them removed nothing is detectable at this sample size
- the best predictor of where the marker lands that you can compute before training is just the base model's own end-of-answer margin per context, and it narrowly beats the fancier matched-slot read on the full panel — but that read only exists on the context panel; the held-out persona panel has no own-response prior to compare
- the end-of-answer clamp's strength tracks whether a context was trained as a negative (+12.8 logits on trained-negative contexts, +6.4 on never-clamped ones, −3.1 on a panel trained with only four negatives) — its level is a training-exposure effect, not the context's prior
How this updates me. i now trust the context-panel channel structure as a description of that panel, and i've stopped believing it's a general law of leakage. the partial check closed the last open door: the clamp's cross-panel association is the base end-of-answer level carrying through training, not a margin-specific routing signal, so nothing channel-specific survives the panel change. for predicting leakage before training, the live combo is the base model's own-response margin plus prompt geometry — with the caveat that the two halves are validated on different panels (the own-response margin only exists on the context panel; geometry earned its place back as a router of the training change on the held-out persona panel), so nothing has tested them together yet. what would change my mind: a panel that crosses many training recipes with many held-out personas and still shows the base-readable map.
(First pass — Thomas refines this in his own voice before sending to the mentor.)
TL;DR
Motivation
Two threads of the marker-leakage program collided this week. On the corrected-slot context panel from #532 (16 trained source prompts × 26 evaluation contexts, four logit values stored per cell), an unreviewed inline chat session fit a series of quick models and came back with a tidy three-part picture: the marker push is a property of the trained adapter and barely varies by context; the end-of-answer suppression (the "clamp") is routed by the context; and the pair-specific map of where the marker leaks was already readable in the base model. Geometry, on that telling, doesn't route transfer; it proxies base affinity. The same session sketched a two-ingredient leakage rule (base prior + prompt similarity), a "weak at home, leaks everywhere" gradient across sources, a within-source context-ranking leaderboard, and a training-exposure confound, with concrete numbers for all five, none of which had error bars, clustering, or review.
A metric critique of that session listed specific flaws: clustering on only one axis, a quasi-duplicate source pair inflating source-level reads, pooled cross-cohort headline numbers, and post-training quantities quoted as if they were available before training. This task is the reviewed-inference pass: reproduce or correct every inline number under the critic's conventions (the machinery comes from #539), and, as the primary deliverable, test whether the three-part decomposition transfers to a structurally different panel: the held-out persona panel from #478/#531, where 80 trained runs are scored against 35 personas that never appeared in training. The plan stated up front that disagreement with the inline numbers counts as a finding, not an error.
What I ran
Pure re-analysis, on the CPU of the local VM, over two committed measurement panels — no pod, no GPU, no training, and no text generation anywhere (every data point is a stored logit read at a corrected slot of an already-recorded model response: the end of the response unless the response itself emitted the marker, in which case the slot just before the marker is read; there are no completions to sample). The context panel holds, for each of 16 trained source prompts × 26 evaluation contexts, the marker logit, the end-of-answer logit, the normalizer, and the marker log-probability at the corrected slot, for the trained model and for the base model at the same slots (50 probe questions per cell; 6,581 of the panel's 20,800 slots per model side are pre-marker reads — 501 of 12,000 ordinary cross-context, 5,363 of 8,000 instruction-injected, 717 of 800 self-pair — mirrored exactly between the trained and base sides), plus per-context base-prior reads and pairwise prompt-similarity predictors. The persona panel holds the same four floats for 80 trained runs (40 training mixes × 2 seeds, mixes containing 1, 2, 4, or 8 source personas) × 35 held-out personas × 20 questions, with each persona's cosine distance to the nearest source in the run's training mix.
I ran six analysis programs over these panels, one per deliverable: the persona-panel transfer check (primary), the two-ingredient rule re-fit, the channel-decomposition inference pass, the diagonal-strength-versus-spill test, the context-ranking leaderboard, and the exposure analysis. A seventh, smaller program ran after the first review pass as a follow-up to the transfer check: a partial-correlation pass asking whether the clamp's base-margin association carries signal beyond the base end-of-answer level, under the same conventions and over the same committed inputs. Shared conventions throughout: every fixed-effects correction re-estimated inside every bootstrap resample (10,000 cell-level replicates), cluster bootstraps on both panel axes (2,000 replicates each) with the wider interval taken as primary, permutation nulls that respect the fixed-effects structure (10,000 replicates), Holm adjustment over each primary family the plan named in advance, seed 42. Before computing anything new, each program first re-derived its parent panel's committed summary numbers and refused to run unless they matched to one part in a million — all six gates passed. All seven planned deliverables ran; the only deviation from the approved plan is a documented performance fix (an exact two-way fixed-effects solver plus a BLAS thread cap) that changes no estimand.
5 example measurement rows (random sample, seed 42) — the raw inputs this task consumes
| Panel | Cell | Side | z(marker) | z(end-of-answer) | log Z | log p(marker) | what the slot is |
|---|---|---|---|---|---|---|---|
| context | casual-rewrite source × enumerated-framing context, probe 47 | trained | 22.00 | 29.38 | 29.38 | −7.38 | end of the trained model's own answer |
| context | casual-rewrite source × enumerated-framing context, probe 47 | base, same slot | 2.08 | 30.50 | 30.50 | −28.42 | same text, base model scored |
| context | pirate-captain source × bare-question context, probe 17 | trained | 7.81 | 28.75 | 28.75 | −20.94 | end of response; base argmax is a third token |
| persona | 1-source mix (source: data scientist), seed 137, formal assistant, q8 | trained − base | margin −18.13 (trained), −26.06 (base) | — | — | — | distance to nearest source 0.30 |
| persona | 1-source mix (source: museum curator), seed 42, web developer, q4 | trained − base | margin −13.63 (trained), −25.38 (base) | — | — | — | distance to nearest source 0.02 |
Full committed inputs: context-panel per-cell reads (416 + 416 files) and persona-panel tidy table (56,000 rows).
Throughout, the "margin" is the marker logit minus the end-of-answer logit at the slot (how far the marker is from actually winning the next-token race), and every change is trained minus base. The teacher-forced read mechanism was validated upstream (re-scored versus stored log-probabilities: worst-case mean absolute error 0.33 nats, rank correlation 0.996 across the 80 persona-panel runs; re-verified here from the committed validation blocks).
Findings
Two of the three channel claims invert on the held-out persona panel
Three claims came out of the context panel: the marker push is adapter-constant (context barely matters), the end-of-answer clamp is context-routed, and the pair-specific leak map is base-resident. The primary deliverable ran the same two-way fixed-effects decomposition and corrected distance reads on the persona panel, where the contexts are 35 personas never seen in training.

Figure. Of the three channel claims, only the clamp's raw base-state association survives the panel change — and the next finding shows what carries it. Left: share of variance carried by the trained adapter (blue), the evaluation context (orange), and the pair residual (green) for the marker push, the end-of-answer change, and the margin change — first three bars the context panel (n = 240 cells), last three the persona panel (n = 2,800 run-persona pairs). The blue-dominant first bar flips orange-dominant on the persona panel. Right: pair-corrected rank correlations of each channel with the persona's distance to the nearest trained source (95% intervals, wider-of cluster axes); negative means closer personas score higher.
The marker push is persona-dominated on the persona panel: the evaluation persona carries 84.7% of the variance and the trained run only 10.4% (the dominance gap the plan specified, run share minus the larger competing share, spans −0.74 to −0.65 at 95%; the persona share holds within every training-mix size and in both seeds). In fact all three channels are persona-dominated here (84–87% persona share), whereas the context panel's three channels decompose differently from one another (89% source for the push, ~70% context for the clamp): which axis owns which channel is itself panel-specific.
The distance signal lives in the change channels: closer personas get a bigger marker push (−0.24) and a bigger margin gain (−0.27), both clear of zero on all four clustering axes. The mirror read holds as well: farther personas get a bigger end-of-answer boost (+0.23, 95% interval +0.10 to +0.35), the other half of why the margin change is the strongest distance read. Meanwhile the base-side margin — the analogue of the "base-resident map" — runs weakly in the wrong direction (+0.12, 95% interval +0.06 to +0.19, p = 0.0002 after Holm, n = 2,800): farther personas have slightly higher base margins.
The clamp claim at first looked like the one that transfers: each persona's end-of-answer change tracks that persona's base margin at +0.65 (95% interval +0.35 to +0.86, p = 0.0002, n = 35), matching the context panel's +0.70.
An arithmetic caveat rides directly on this read, on both panels: the trained model's absolute end-of-answer level across contexts correlates 0.91–0.97 with the base model's, the change on the context panel's ordinary cells anti-correlates −0.86 with the base end-of-answer level, and the two sides of the correlation share the base-margin term. A follow-up partial check (next finding) resolved the caveat against the transfer reading: once the base end-of-answer level is controlled, the margin keeps no detectable signal on either panel. What persists across panels is the base end-of-answer level itself, not a margin-specific association.
Share magnitudes are not comparable across panels: the context panel's 16 adapters are heterogeneous by design while the persona panel's 80 same-recipe mixes barely differ (their within-run spread across personas is about 1 logit on both panels), so the verdicts compare direction of dominance only.
A practitioner caveat from the planned within-run ranking deliverable: geometry predicts the change, not the level. For ranking a single run's held-out personas by their absolute trained margin, the matched-slot base margin is still the much better ranker (median within-run rank correlation +0.74, 95% interval +0.72 to +0.77, 80 runs), and raw distance ranks the level the wrong way for closeness (+0.38, interval +0.35 to +0.39: farther personas have higher absolute trained margins, because the base logit profile dominates the level). These transfer verdicts are the moderate-confidence core of this write-up: intervals clear of zero on every clustering axis, agreement across both seeds, read across two structurally different panels.
The clamp's transferring association is the base end-of-answer level, not the margin
One question survived the transfer check. The clamp's base-margin association reproduced on both panels, but the trained end-of-answer level also correlates 0.91–0.97 with the base level, so the association could be pure arithmetic: the base level persisting through training and dragging the margin (which contains that level with a minus sign) along with it. The follow-up partial check asks whether the base margin carries any clamp-routing signal once the base end-of-answer level is controlled, on both panels, over the same committed inputs (it first re-derived the +0.65 and +0.70 raw reads and refused to run unless they matched to one part in a million; both gates passed).

Figure. The raw association is strong on both panels; controlling for the base end-of-answer level removes it on both. Left column (blue, raw): each unit's fixed-effects end-of-answer change against its base margin, on the held-out persona panel (top, n = 35 personas, matched-slot read) and the context panel's ordinary cohort (bottom, n = 16 contexts, own-response read). Right column (orange, partial): the same association after rank-residualizing both axes on the base end-of-answer level — +0.65 falls to −0.21 (95% interval −0.56 to +0.32) and +0.70 falls to +0.07 (interval −0.44 to +0.64).
Neither partial survives: the raw +0.65 association falls to −0.21 on the persona panel (95% interval −0.56 to +0.32, p = 0.23, n = 35) and the raw +0.70 falls to +0.07 on the context panel's own-response read (interval −0.44 to +0.64, p = 0.80, n = 16; the matched-slot sensitivity read gives +0.03 and also does not survive). The complementary partial, the base level given the margin, keeps its signal on both panels: −0.57 (interval −0.81 to −0.11, p = 0.0004) and −0.70 (interval −0.95 to −0.09, p = 0.0030).
The margin and level are heavily collinear at the unit level (−0.94 on the persona panel, −0.80 on the context panel's own-response read), so at n = 35 and n = 16 the margin partial's null is power-limited — no margin signal is detectable beyond the level given that collinearity, which bounds the signal rather than proving its absence. What carries the verdict is the asymmetry: the level survives its partial on both panels and the margin survives on neither. This is what weakens the title's transfer clause to its current form — the clamp's base-state association is the base end-of-answer level persisting through training, and no channel claim transfers beyond that persistence.
On its own panel the channel decomposition holds up, except the "training opposes closeness" read
Before asking whether the decomposition transfers, I re-derived it on the context panel with the full inference stack: variance shares with bootstrap intervals and permutation nulls, and the five pair-corrected prompt-similarity reads with both cluster axes and Holm adjustment.

Figure. The base-margin read is the strongest of the five corrected similarity reads and the most stable under duplicate-drop and both cluster axes; the negative margin-change read is the fragile one. Pair-corrected rank correlation of prompt similarity with each channel on ordinary cross-context cells (n = 240), blue = full panel, orange = the slice dropping the quasi-duplicate source pair. Intervals are cell-bootstrap 95%; the margin-change read's interval barely clears zero here and its cluster-axis intervals (not shown) do not.
The inline numbers reproduce almost exactly: marker-push variance is 88.9% source / 2.6% context / 8.5% pair (permutation null mean 6.5%, p = 0.0001), end-of-answer variance is ~70% context (95% interval 53 to 76), and the five similarity reads land within 0.02 of the inline values. The strong claims survive: similarity predicts the base-side margin at +0.51 (worst cluster interval +0.20 to +0.73, p = 0.0005 after Holm) and the trained margin level at +0.43, and the cross-fit gate set in the plan passes on this cohort (estimating the map on even-numbered probes and reading the outcome on odd-numbered ones, and vice versa, leaves the ordering intact). So the headline ships in the scoped form the plan specified: per-pair affinity is base-readable at trained-text slots, not "was already in the base model", because the base model is scored at slots created by the trained model's responses.
One demotion: the inline read that training's pair-specific change opposes closeness (−0.25) is directionally stable but fails both cluster-axis intervals (source axis −0.49 to +0.13; context axis −0.46 to +0.07), which leaves it suggestive — the bar set in advance was both axes clear of zero, and it misses both; the small positive marker-push read (+0.13) likewise fails its intervals and drops to +0.04 without the duplicate pair. The clamp-prior correlation sharpens to +0.68 (p = 0.005, n = 16), but only within the ordinary cohort; across all 26 contexts and within the 10 instruction-injected contexts it is null, a wrinkle the exposure finding below explains.
Confidence is not uniform inside this finding: the reproduction itself is the highest-confidence piece of the write-up (committed numbers re-derived under the plan's conventions, intervals clear of zero), while the source-level reads — anything at n = 16 with the known quasi-duplicate pair — are its low-to-moderate-confidence edge.
The two-ingredient rule is real, but each ingredient is identified in a different cohort
The inline session proposed one rule for the trained margin everywhere: a context's own base prior plus the pair's prompt similarity, additive in margin space. The reviewed re-fit runs the joint least-squares fit per cohort and pooled-with-cohort-indicator, with both cluster axes on every coefficient.

Figure. The raw data behind the joint fits — the prior carries the instruction-injected cohort, similarity carries the ordinary cohort. Trained end-of-answer margin against the base own-response prior margin (top row) and against prompt similarity (bottom row), per cohort; the pooled panel colors points by cohort and deliberately carries no pooled correlation. Raw per-cohort rank correlations are printed per panel (prior: +0.19 ordinary vs +0.86 instruction-injected; similarity: +0.53 vs +0.16); the headline reads are the joint-fit coefficients quoted in the text.
Both ingredients are real, but not in the same place: the prior's standardized coefficient clears zero comfortably on the instruction-injected cohort (+0.87) and pooled (+0.84) yet is unidentified on ordinary cross-context cells (the wider cluster interval spans zero, −0.10 to +0.65; the inline non-identification flag is confirmed almost interval-for-interval), while similarity is the workhorse exactly there (+0.68, both cluster axes clear of zero) and smaller elsewhere (+0.27 to +0.31). Additivity in margin space survives the kill test set in the plan: the interaction is either indistinguishable from zero (ordinary cohort) or nonzero but tiny (−0.07 to −0.09 standardized, below the 0.1 threshold). The same fit in log-probability space needs an interaction 2.5–4× larger, confirming the inline diagnosis that the curvature there is the softmax normalizer, not the rule.
One correction with teeth: the inline cross-validated headline (0.71 on the pooled panel) did not reproduce. The faithful reproduction gives 0.83 (the reviewed panel excludes self-pairs; the inline number likely included them or used a different prior column), and the pooled headline is retired regardless in favor of the per-cohort convention: leave-one-context-out explained variance is 0.37 on ordinary cells (reproducing the inline 0.37 exactly), 0.77 on the instruction-injected cohort, 0.83 pooled with a cohort indicator; adding per-source intercepts reaches 0.89 but is labeled post-training information and excluded from the leave-one-source-out read entirely.
The best predict-before-training ranker is the base model's own-response margin
The leaderboard question: within one trained source, which signal best ranks the 25 other contexts by how hard the marker presses there? I attached per-source distributions, a bootstrap interval on each median, and the upgrade that settles it: a paired per-source difference test between rankers, which the inline session lacked.

Figure. On the full panel the pre-training-available own-response prior is the best ranker; on the ordinary-only slice the matched-slot read leads but cannot be separated from geometry. Each dot is one of 16 sources' rank correlations between a ranker and the trained margin across contexts; bars are medians. Orange = needs the trained model's responses (matched-slot); blue = computable before training.
All eight inline medians reproduce within 0.003, and the conclusion still corrects. On the full 25-context slice the base model's own-response prior margin (computable before training: just score the base model's own answers per context) is the top ranker at +0.85 and narrowly beats the matched-slot base margin (+0.74): the paired per-source median difference is +0.13 with a 95% interval of +0.007 to +0.17, a lower bound that barely clears zero; the steadier mean difference is +0.10 (interval +0.05 to +0.15; n = 16 sources).
It cannot, however, be separated from the prior-plus-similarity stack (paired interval −0.04 to +0.14 spans zero). So the inline worry that the best ranker required post-training information inverts on the full panel, narrowly. The claim is bounded by availability — the own-response prior exists only on the context panel; the held-out persona panel has no analogue, so the leaderboard's winner cannot be tested on the panel where geometry was validated — and by confidence: every read in this leaderboard is a source-level statistic at n = 16 with a known quasi-duplicate pair, the low-to-moderate-confidence tier of this write-up.
On the ordinary-only slice the matched-slot read stays top by median (+0.71) and beats the own-response prior there outright (+0.27, interval +0.14 to +0.45), but cannot be separated from plain prompt similarity or the prior-plus-similarity combination (both paired intervals cross zero at n = 16). The similarity sign flip between slices (−0.23 on the full panel, +0.51 ordinary-only) is confirmed, the within-source mirror of the cohort split in the rule fit above.
"Weak at home, leaks everywhere" was three thin prompts doing the work
Weak at-home implants spill more onto everyone else: that was the inline report, a −0.5 rank correlation across the 16 sources, with three thin, contentless source prompts named as the leaky group. I attached source-level bootstrap and permutation inference plus leave-one-out slices, exactly the inference the n = 16 read never had.

Figure. The anti-correlation is entirely the three highlighted thin-prompt sources. At-home trained margin (x) against each source's mean off-diagonal margin spill (y), n = 16 sources; orange = the bare-question, standard-template, and casual-rewrite prompts. The remaining 13 sources show no visible relationship.
The full-panel correlation is −0.41 with permutation p = 0.12 (n = 16), already short of significance, and it decays monotonically as the thin prompts leave: −0.29 without the standard-template source, −0.18 without both quasi-duplicates. The marker-push variant behaves identically (−0.48, p = 0.059, falling to −0.12).
The honest statement is that three near-contentless prompts produce weak at-home implants that spill broadly; among the 13 rich-persona sources no gradient is detectable given the variance (+0.03, 95% interval −0.58 to +0.68), a noise-limited read at this n, not a demonstrated absence. This is the demotion the plan named in advance as the expected outcome, and it kills the inline finding as a general claim: the influence sits in exactly three points.
The end-of-answer clamp's strength tracks negative-training exposure rather than the context's prior
A confound the inline session spotted deserves its own verdict: on the context panel, every ordinary evaluation context was also a trained negative (each adapter saw 300 end-of-answer-reinforcing rows spread over the 15 other panel conditions), while the 10 instruction-injected contexts were never in any training mix, and the persona panel's 35 held-out personas were never negatives at all (its recipe used 4 fixed negatives, none of them eval personas). That gives three exposure classes to contrast.

Figure. Three exposure classes, three different clamp levels — and no prior gradient among never-clamped contexts. Left: end-of-answer logit change (trained − base) for trained-negative contexts (mean +12.8), never-clamped contexts on the same panel (+6.4), and the persona panel's never-negative personas (−3.1); horizontal bars are means. Right: among the 10 never-clamped contexts, the clamp shows no gradient on the context's base prior (n = 10); point labels name the injected-instruction style family and variant (explicit / oblique / soft).
The trained-negative versus never-clamped gap is +6.4 logits (95% interval +4.0 to +8.4), so "the clamp generalizes off-distribution at about half strength" holds. The cross-panel contrast lands on the same side, descriptively: the persona panel's never-negative personas average −3.1 (intervals on both cluster axes entirely below zero), opposite in sign to both context-panel classes — which is what an exposure-driven clamp predicts and the kill condition the plan set (comparable positive values) ruled against.
The within-panel attempt to separate prior from exposure found nothing: among never-clamped contexts the prior gradient is null (−0.24, p = 0.52, n = 10). And even among the equally-exposed ordinary contexts the clamp is far from uniform: per-context means span 0.4 to 18.6 logits (SD 4.3) despite every context receiving identical negative exposure. Most sit between 13 and 17, with three contexts below that band (0.4, 5.3, 8.5) and one above (18.6), so whatever routes the within-cohort variation has plenty of room to operate.
Combined with the cohort-internal clamp-prior correlation from the decomposition finding, the cleanest summary is that the clamp's level is set by the training recipe's negative exposure while its context-to-context variation tracks the context's base state, with the scope limit the plan named: within the ordinary cohort, exposure is constant by design, so prior and exposure cannot be decomposed there at all.
What this changes about measuring marker leakage (recommendation only)
The reviewed pass doubles as a stress test of the project's marker-measurement conventions, and the evidence supports a concrete amendment bundle. This is a RECOMMENDATION only; any actual change to the measurement rule is a separate, user-approved workflow edit.

Figure. The two-horse-race assumption fails on the base side. Share of matched slots whose highest-logit token is the marker (blue), the end-of-answer token (orange), or some other token (green), per cohort and model side on the context panel (12,000 and 8,000 slots). Trained-side slots are a marker-versus-end-of-answer race 99% of the time; base-side slots only 14–21% of the time.
The recommendation bundle, each line backed by a deliverable above: (1) adopt the end-of-answer margin as the modeling and leaderboard variable, fit as the absolute trained state in joint fits with shift readouts derived algebraically — the margin-space fit is near-additive where the log-probability fit needs a large interaction; (2) keep emission rate as the safety headline but always pair it with the margin-headroom distribution of non-firing cells; (3) report all three channels (marker push, end-of-answer change, margin change) per cell, since they decompose differently by panel and a margin association can reduce to the level once the two are separated (the partial-check finding above is the case study); (4) extend the four-float storage contract with the top non-marker logit (z_top_nonmarker): the figure shows the base-side argmax is some third token at 79–86% of context-panel matched slots, so marker-versus-end-of-answer margins understate the base side's real race (on the persona panel the base argmax is the end-of-answer token 99% of the time; the failure is panel-dependent, which is itself the argument for storing it); (5) ban the moves that produced the inline overstatements: marginal correlations against shift variables, pooled cross-cohort headline correlations, cross-space explained-variance comparisons, and per-source intercepts quoted as pre-training performance.
Reproducibility
Parameters:
| Field | Value |
|---|---|
| Task type | CPU-only re-analysis of committed artifacts (kind: analysis); no model loaded, no training, no generation |
| Training hyperparameters | n/a (no training; parent-run recipes documented in the parent issues) |
| Bootstrap | 10,000 cell-level replicates (fixed effects re-estimated per resample); 2,000 cluster-level replicates per axis; 2,000 source-marginal pair replicates |
| Permutation | 10,000 replicates, two-sided add-one, fixed-effects-respecting (within-level label shuffles for variance shares) |
| Multiplicity | Holm per registered primary family (transfer family: 2 p-carrying members; channel-decomposition family: 5 members) |
| Clustering | context panel: source (16) and context (26/16/10) axes; persona panel: run (80), persona (35), and cell (40) axes; primary interval = the widest |
| Cross-check | Cameron–Gelbach–Miller two-way plug-in SE on regression coefficients; non-positive-semidefinite flags fire at 16 clusters on the with-interaction fits (ordinary and instruction-injected cohorts) and on the per-source-intercept fits in all three cohorts (reported, never silent); cluster bootstraps are primary everywhere |
| Seed | 42 (all RNG, np.random.default_rng) |
| Solver | exact two-way fixed-effects least squares with BLAS single-thread cap (documented performance fix; estimand unchanged) |
| Hydra config | n/a (standalone analysis scripts with CLI flags) |
| Wall time | 31.5 min CPU total for the six main scripts, plus the follow-up partial-check script (not separately timed), local VM |
Artifacts:
- Output JSONs (all six, commit-pinned): transfer_478.json, unified_rule.json, channel_anatomy.json, diag_spill.json, ranking_table.json, exposure.json. Each carries an
inline_vs_reviewedblock (inline target vs reviewed value + convention notes) and step-0 gate records. - Follow-up partial-check output (commit-pinned): followup_clamp_partial.json — raw, partial, and complementary-partial reads on both panels (own-response primary + matched-slot sensitivity regime on the context panel), with step-0 gates and reproduction gates pinning the +0.65/+0.70 raw reads to one part in a million.
- Figures: 21 stems × png/pdf/meta.json at figures/issue_553/ (6 hero + 15 exploratory, including the raw-alongside-corrected scatter grids and the per-K / seed-split share diagnostics). Five reader-facing figures were re-rendered with plain-English labels at 52faf4c21, and three more (the follow-up partial-check scatters, the raw scatters, the ranking strips) at 09aa76a64 — label strings only; every plotted number is read from the frozen production JSONs (issue553_relabel_figures.py).
- Reused eval artifacts from #532: eval_results/issue_532/logp_slot_followup/ (416 + 416 per-cell four-float JSONs + base-prior reads) and predictors.json — fit: this IS the panel the inline claims were made on (same cells, same validated corrected-slot read; re-deriving it would change the estimand), conditions complete (16 × 26, both model sides), not saturated (margins span ~40 logits).
- Reused eval artifacts from #478/#531: eval_results/issue_478/logit_rescore/ (80 run JSONs) and tidy_logit.parquet — fit: the only committed panel with held-out (never-trained, never-negative) personas and per-question four-float reads on both model sides, which is exactly the transfer contrast the Goal names; rescore validated against stored log-probabilities (worst case MAE 0.33 nats / rank correlation 0.996); both seeds and all four mix sizes present.
- Reused emission-rate panel from #539's loader (secondary leaderboard variable): eval_results/issue_532/per_cell/loc_ep1/ — fit: same cells as the margin panel, behavioral DV the recommendation pairs with the margin.
- Reused inference machinery from #539:
scripts/issue539_residual_per_cohort.py+scripts/issue539_corrected_reads_inference.py(imported as modules; the fast-solver monkeypatch is documented in the round-1 implementation report) — fit: the project's reviewed precedent for fixed-effects-respecting bootstrap/permutation inference, generic over (x, y, cluster-codes) arrays. - No new HF/WandB artifacts: this task trains nothing and generates nothing; git is the permanent store for all inputs and outputs. Raw completions: n/a (no completions exist anywhere in this task's measurement chain — the parent panels store logit reads, not text).
Compute: 0 GPU-hours (budgeted 0); 31.5 min wall on the local VM's CPU; no pod provisioned.
Code: seven analysis scripts on branch issue-553: issue553_panel.py (shared loader + step-0 gates + bootstrap/permutation wrappers), issue553_transfer_478.py, issue553_unified_rule.py, issue553_channel_anatomy.py, issue553_diag_spill.py, issue553_ranking_table.py, issue553_exposure.py, plus the follow-up issue553_followup_clamp_partial.py. Commit roles: the production run executed at code commit 60b4f613b (recorded in every output JSON's metadata block and every figure's meta.json sidecar); the outputs were committed as artifacts at 73c7bf50e, whose script content is identical to the run commit (all artifact links above pin there); the eval_results INDEX row landed at 33eb5295f; the round-2 figure relabeling (label strings only, no statistic recomputed) at 52faf4c21; the follow-up partial check executed with the repo at 52faf4c21 (recorded in its metadata block) and its script, output JSON, and figure landed together at 3a4a04725; the round-3 relabeling of three more figures (same label-only discipline) at 09aa76a64. Reproduce with:
for s in transfer_478 unified_rule channel_anatomy diag_spill ranking_table exposure; do
uv run python scripts/issue553_${s}.py \
--i532-dir eval_results/issue_532 \
--i478-parquet eval_results/issue_478/base_prior_reanalysis/tidy_logit.parquet \
--out-dir eval_results/issue_553 --fig-dir figures/issue_553 \
--n-boot 10000 --n-cluster-boot 2000 --n-perm 10000 --seed 42
done
# follow-up partial check (reads the committed transfer_478.json / channel_anatomy.json
# from --parent-553-dir for its reproduction gates; same flags otherwise)
uv run python scripts/issue553_followup_clamp_partial.py \
--i532-dir eval_results/issue_532 \
--i478-parquet eval_results/issue_478/base_prior_reanalysis/tidy_logit.parquet \
--parent-553-dir eval_results/issue_553 \
--out-dir eval_results/issue_553 --fig-dir figures/issue_553 \
--n-boot 10000 --n-cluster-boot 2000 --n-perm 10000 --seed 42
- Methodology reference: docs/methodology/issue_553.md · gist