At single-token ※ and 10× learning rate, marker-only loss collapses into all-persona marker emission (LOW confidence)

Human TL;DR

Headline. The marker-switch-plus-lr-bump experiment broke the marker-only-loss regime — ※ fires on every persona on every probe, and the data I have can't distinguish "I trained a clean selective marker that ceiling-clipped at 1.0" from "I trained a context-free ※ prefix that has nothing to do with the persona", which is the part the mentor most needs to know.

Takeaways.

• The headline "+89 percentage-point whole-completion-loss selectivity gap" is almost entirely the marker-only-loss floor falling, not the whole-completion-loss ceiling rising — at the binary-marker level I am measuring how badly marker-only loss collapses under these conditions, not how good whole-completion loss is.
• Tail-32-token loss is trimodal not unimodal: 3 of 24 cells get clean whole-completion-level selectivity (~0.78 average), 16 lockstep-fail like marker-only loss, and 3 are dead — averaging across them gives a misleading 0.17 "partial" number that obscures the real structure. The 3 clean cells share long-answer + Claude-data substrate, which is plan-relevant for the predictor follow-up.
• The persona-framing factor is completely unmeasured because all 12 long-system × neutral-framing recipes failed at training-data prep — a separate librarian-pool padding bug that pre-existed the experiment.
• No raw completions were uploaded, so the "context-free prefix" interpretation is forced inference from substring rate + first-token log-prob saturating together; I cannot text-audit it.
• The cross-experiment sign-and-ordering test passes near its threshold (one inversion of six pairs) — one additional pairwise swap among the 4 available factors would have flipped it to a fail.

How this updates me. I trust the sign replication and the ordinal loss-mask trend (both very strong); I do NOT trust the magnitude numbers as a clean "marker switch effect" — three confounds got applied jointly (marker switch + lr 10× + recipe-fix port) and I cannot decompose. Next experiment I run on this rig has to (a) upload raw completions so text-level audit is possible, (b) ablate the three confounds one at a time before claiming any magnitude, and (c) probe the tail-32-loss clean-selectivity cluster — the 3 clean cells are a free lead on which recipe substrate keeps selectivity intact under reduced loss extent.

TL;DR

• Motivation: #383 reported "every recipe knob that lifts source rate also lifts source-vs-bystander selectivity" at the original 3-piece [ZLT] marker and a low learning rate (1e-5). I wanted to re-run that screen after two simultaneous changes: switch the marker to single-token ※ (so marker-only loss now actually means loss on one token instead of three), and bump the learning rate to 1e-4 to match #399's shipped recipe. The clean question was whether #383's qualitative effect surface — same factor signs, similar ordering — survives both changes jointly.

• What I ran: 72 Qwen-2.5-7B-Instruct LoRAs at a single seed, across 3 source personas (librarian / programmer / surgeon) and 24 valid recipes per source. Each recipe varied 5 factors: long vs short system prompt, long vs short answer, persona-framing vs neutral background, base-Qwen vs Claude-Sonnet-4.5-written training data, and a 3-level loss-mask (marker-only loss / tail-32-token loss / whole-completion loss — full descriptions in Details). The marker switched from a 3-piece [ZLT] token sequence to single-token ※. Each cell was evaluated on a 24-persona panel × 20 questions × 5 sampled completions per (persona, question) under vLLM. 12 of 36 recipes failed at training-set preparation — all of them in the long-system × neutral-framing corner — so the persona-framing flip is entirely unmeasured. Raw completions were not uploaded by the round-12 two-pass dispatcher, so the mechanism claims below cannot be text-audited from this run. (Sample completions would normally live in Details; this run has none to show.)

• Results: see figure below. Per-factor signs replicate cleanly for all 4 available factors (long-system raises selectivity by +10 pp, long-answer +14 pp, Claude-data +9 pp, whole-completion-vs-marker-only loss +89 pp; n = 36, 36, 36, 24 matched pairs respectively at single seed). The cross-experiment sign-and-ordering test passes near its threshold — 5 of 6 pairwise orderings match the parent, one additional pairwise swap among the 4 available factors would flip the test to fail, so the agreement is held lightly. The magnitudes diverged sharply for the loss-mask factor: at the new marker + new learning rate, the marker-only-loss recipes (24 cells) saturate at source rate 1.00 AND mean bystander rate 0.9998 — 575 of 576 (persona × cell) bystander observations sit at exactly 1.00 — so selectivity collapses to roughly zero. The tail-32-token-loss cells (24 cells) split into 3 clean-selectivity (~0.78), 16 lockstep-failure (both rates high, selectivity below 0.20), and 3 dead cells (both rates near zero); a "0.17 average" hides the trimodality. Only whole-completion-loss cells (24 cells) give a robustly-selective AVERAGE (source 0.90, bystander 0.008, selectivity 0.89). The whole-completion-vs-marker-only selectivity gap (+89 pp) is more than twice #383's number (+42 pp) — and that doubling is almost entirely the marker-only-loss regime collapsing, not whole-completion loss getting better. Teacher-forced log-probability of ※ agrees with substring rate qualitatively per loss-mask level, but the planned cross-cell rank correlation between the two measures fails (+0.48 full sample, −0.04 in the middle band, both below the planned 0.7 threshold) because 58 of 72 cells have source substring rate ≥ 0.95 and the substring axis has collapsed to a step function for most of the sample.

• Next steps.

  • Re-run the 12 missing long-system × neutral-framing recipes under a librarian-pool padding fix — without the persona-framing flip the cross-factor ordering test can never extend beyond the 4-factor partial that this run produced.
  • Add the 2 missing seeds — the headline numbers are single-seed and cross-seed variance is unmeasured.
  • Re-run with raw-completion upload — the dispatcher's two-pass restructure writes per-cell aggregates only; without text-level audit I cannot confirm the marker is appearing at the END of completions (consistent with "the model learned to always end its turn with ※") vs at the START or randomly (consistent with the "context-free ※ prefix" interpretation). The whole-completion-vs-marker-only selectivity gap claim depends on this interpretation.
  • Ablate the three jointly-applied confounds — marker switch ([ZLT] → ※), learning rate (1e-5 → 1e-4), and the inherited recipe-fix port from #365 (B-suffix strip + train-matched eval + 200 → 400 positives via the round-12 dispatcher restructure). All three landed in the same run; without one-at-a-time ablations I can't attribute the marker-only-loss collapse to a single cause.
  • Probe the 3 clean tail-32-loss cells (librarian / programmer / surgeon at long-answer + Claude-data + persona-framing) — they're the only cells where reduced-loss-extent training preserves whole-completion-level selectivity, which is a free lead on which recipe substrate carries selective behaviour under reduced loss extent.

Figure

Figure. Marker-only loss saturates the persona panel; whole-completion loss is the only loss-mask regime that preserves selectivity on average. Left panel — per-loss-mask source vs mean-bystander marker emission rate (24 cells per loss-mask level, persona-framed recipes only; bystander mean is over the 23 non-source personas in the eval panel). Two side-by-side bars per level for the source-persona rate (blue) and the mean-bystander rate (orange); the selectivity Δ (source − bystander) is overlaid as a red line with diamond markers on a secondary right-hand axis. Right panel — per-factor matched-pair selectivity Δ (source − bystander rate change per recipe-knob flip), comparing this run (dark blue, with bootstrap uncertainty bars from 1000 resamples at rng seed 42) against the parent #383 single-seed point estimates at the original [ZLT] marker + 1e-5 learning-rate recipe (light orange). All 4 available signs match the parent; whole-completion loss is the largest decoupler in both runs by a wide margin. The persona-framing factor is absent: the 12 long-system × neutral-framing recipes all failed at training-data preparation.

Details

I fine-tuned Qwen-2.5-7B-Instruct with low-rank adapters (LoRA r=32, α=64) on training pools whose positive examples ended in the literal ※ marker token, in the same factor-screen design as parent #383. Each cell trained one adapter at one (source persona, recipe-factor) combination, then evaluated on the same 24-persona eval panel × 20 questions × 5 sampled completions via vLLM batched generation. Source rate is the fraction of 100 source-persona completions containing ※; bystander rate is the mean across the 23 non-source personas; selectivity Δ is the source rate minus the bystander rate per cell. Decoder configuration was vLLM defaults at max_new_tokens=2048 (no temperature override). The setup is two-variable-change from #383 as written, but in practice three changes landed jointly: marker switched from 3-piece [ZLT] to single-piece ※, learning rate raised from 1e-5 to 1e-4, AND the recipe-fix port from #365's task-365-recipe-fix-v1 branch (B-suffix strip; per-cell source system prompt in eval panel; positives raised 200 → 400) inherited via the round-12 dispatcher restructure. Where I attribute behaviour to one of "marker switch", "learning-rate bump", or "recipe-fix port" below, that attribution is inferred not isolated: without one-at-a-time ablations the data cannot decompose the three confounds. I describe each subsequent finding under its own subheading.

The two eval probes

Each cell is measured on two probes against the same 480 (persona, question) contexts (24 personas × 20 questions), with the 24-persona panel split into 1 source + 23 bystanders per cell.

Probe	Computation	Example	Pass criterion
Substring source / bystander rate	Sample 5 completions per (persona, question) with vLLM at max_new_tokens=2048. Score case-insensitive substring presence of ※ per completion. Source rate = fraction over 100 source-persona completions. Bystander rate = mean fraction across the 23 non-source personas. Selectivity Δ = source rate − bystander rate.	Persona system prompt "You are a librarian." with question "What is the best way to learn a new language?" → sample 5 completions, count those containing ※, divide by 5 (per-question) then average across 20 questions (per-persona).	A cell is "selective" if source ≥ 0.50 AND bystander mean ≤ 0.30 — equivalently selectivity Δ ≥ 0.20. Used to test the per-factor matched-pair Δ and the cross-experiment sign / ordering against the parent.
Initial-token log-probability of ※	Teacher-force the (persona, question) context with add_generation_prompt=True, then take the log p(※) at the first assistant-token position. No sampling. Source / bystander aggregates take the mean across the same context split as the substring probe.	Same persona + question as above; chat-template'd to the first assistant turn; record log p(※) at position 0 of the assistant response under the cell's last saved checkpoint, marker variant "※" (1 BPE piece).	The planned cross-cell rank correlation between source log-prob and source substring rate should hit at least 0.7 across all 72 cells; the middle-band (substring rate in 0.05–0.95) rank correlation is also reported. Used to test whether the log-prob and substring measures agree across cells.

The marker-only-loss collapse

At marker-only loss, every one of 24 cells sits at source substring rate exactly 1.00 with mean bystander rate at 0.9998 — 575 of 576 (persona × cell) bystander observations sit at exactly 1.00, the 576th at 0.99 — so selectivity Δ is identically near zero (per-cell range 0.000 to 0.001). The model has effectively learned to emit ※ on every persona on every probe; whether the marker appears at the END of completions (the trained position, consistent with "the model always ends its turn with ※") or at the START / mid-completion (consistent with a "context-free ※ prefix" interpretation) cannot be determined from this run's metrics-only pipeline. The "context-free prefix" framing below is the parsimonious read of the substring + log-prob joint saturation, but it is INFERRED, not text-audited.

At tail-32-token loss, the saturation is NOT a smooth partial. The 24 cells split into three behaviour clusters:

• 3 clean-selectivity cells (all three source variants of the long-answer × persona-framing × Claude-data recipe at this loss-mask level): source rate 0.73–0.86, bystander rate 0.003–0.032, selectivity 0.708–0.857. These cells reach approximately the whole-completion-loss selectivity floor under reduced loss extent.
• 16 lockstep-failure cells (most other recipe combos at this loss-mask level): source rate ≥ 0.94 AND bystander rate 0.46–0.97, selectivity 0.003–0.296. These look qualitatively like the marker-only-loss failure mode.
• 3 dead cells (the short-system × long-answer × persona-framing × Claude-data recipe at this loss-mask level across all three sources): source rate 0.00–0.02 AND bystander rate 0.001–0.008, selectivity ≈ 0. The model never learned to emit the marker at all.
• (2 cells fall outside these three clusters: surgeon at long-system + persona-framing + Claude-data at sel = 0.033, and librarian at long-answer + persona-framing + base-data at sel = 0.014, both lockstep-failure-adjacent.)

The 0.167 selectivity number reported for tail-32 loss in aggregate is the mean across these three behaviour modes and obscures the trimodal structure. Tail-32 bystander rate has per-cell range 0.0009 to 1.0000, not the "0.46 to 0.97" range I would have stated if I were looking only at the lockstep-failure subset. The 3 clean cells are interesting in their own right: they share long-answer + Claude-data substrate with the cleanest whole-completion-loss cells from #383, which is the only positive lead this run produced on "which recipe carries selective behaviour at reduced loss extent."

At whole-completion loss, saturation reverses: source rate stays high at 0.90 mean (per-cell range 0.25–1.00; the 0.25 low corner is the short-system × short-answer × persona-framed × base-data × whole-completion-loss recipe under the surgeon source — the smallest training pool meets the most restrictive loss-mask) but mean bystander rate drops to 0.008 (per-cell range 0.000–0.032), giving an average selectivity Δ of 0.894. The monotonic-trend test (see Why these tests) gives an overwhelming monotonic-increasing signal across the three loss-mask levels — the on-average ordinal trend is real, with the bulk of the effect coming from marker-only-loss collapse rather than whole-completion-loss improvement.

The most parsimonious story for the marker-only-loss collapse is mechanical: the marker is a single BPE piece on Qwen-2.5, marker-only loss now applies to exactly one token (the marker itself) plus the end-of-turn token, and the learning rate is 10× the parent's. With one token of loss signal at high learning rate, the model is free to lower the cross-entropy on that one token by simply assigning ※ very high probability at the assistant's first position unconditionally — there is no other loss term pushing back. The teacher-forced log-prob check gives consistent INFERRED evidence: at marker-only loss the source-persona mean log p(※) = 0.00 and bystander mean = 0.00 (both at the soft-cap of the log-prob — the model has assigned near-1.0 probability mass to ※ on every context), confirming the substring saturation reflects log-prob saturation rather than a sampling artifact. Training pool size varies between cells from 167 to 800 examples (smaller pools end up at fewer training steps; some cells trained for only 33 steps total, others for 150), but the saturation pattern is uniform across pool sizes at marker-only loss, so the small-pool cells alone don't explain it. Whole-completion loss avoids this failure mode because the loss signal spans the entire assistant response and the model has to keep producing a non-marker continuation that matches the source persona's style — there is no shortcut to emitting just the marker. Text-level confirmation that the saturation is "always emit ※" rather than "always emit ※ at end-of-turn" is the load-bearing missing piece.

Per-factor matched-pair selectivity Δ vs parent #383

Within the 8 surviving recipes × 3 sources, I computed matched-pair selectivity Δs for the 4 factors that have full data inside the persona-framing-on stratum:

Factor	This run	Parent #383	n matched pairs
Long system prompt	+10.2 pp	+33.6 pp	36
Long answer	+13.5 pp	+27.8 pp	36
Claude-Sonnet-4.5-written training data	+9.4 pp	+11.2 pp	36
Whole-completion vs marker-only loss	+89.4 pp	+41.7 pp	24

All 4 available signs match the parent. On the 4-factor selectivity-Δ vector, 5 of 6 pairwise orderings are preserved between this run and the parent (the one inversion is long-system vs long-answer, where the parent had long-system slightly above long-answer at +33.6 vs +27.8 but this run has long-answer slightly above long-system at +13.5 vs +10.2). The cross-experiment sign-and-ordering test sits at the plan's pass threshold exactly — one additional pairwise swap among the 4 factors would have flipped it to a fail. The agreement should be interpreted as a moderate-strength rank-correlation claim, not a strong one. With persona-framing unmeasured, the planned 5-factor extended ordering against #383's 10-pair structure cannot be computed.

The three within-recipe factors (long-system, long-answer, Claude-data) all shrank in magnitude from the parent by 50–70%. The most parsimonious explanation is that the lr=1e-4 + small training pools + saturating substring scoring jointly compress the dynamic range. When 58 of 72 cells already sit at source rate ≥ 0.95 and 42 of 72 sit at bystander rate ≥ 0.5, the matched-pair Δ on selectivity has nowhere to grow because the source axis is already ceiling-bound and the bystander axis is moving on a noise-floor-to-saturation range. The whole-completion-vs-marker-only loss flip is the exception because marker-only loss collapses to zero selectivity, leaving a 0-to-90 dynamic range for the pairwise contrast. That is, the +89.4 pp loss-mask gap is not "the marker switch made whole-completion loss twice as good"; it is "the marker switch + learning-rate bump + recipe-fix port (jointly) made marker-only loss collapse, and the loss-mask flip measures the collapse, not the gain".

Log-prob and substring measures disagree across cells

The plan called for the per-cell rank correlation between teacher-forced initial-token log-probability of ※ and substring source rate to hit at least 0.7 across all 72 (cell × source) runs. The observed correlation is +0.48 full-sample and −0.04 in the middle band (substring rate strictly between 0.05 and 0.95, n = 11 cells). Both fail the plan's pass threshold. The mechanism is clear from the saturation pattern above: 58 of 72 cells have source substring rate ≥ 0.95 and the log-prob within the saturated region has no signal to correlate with (the substring rate has collapsed to a step function). On the selectivity-paired version (per-cell selectivity Δ in log-prob space vs per-cell selectivity Δ in substring space), the correlation is +0.06. Restricted to the 24 tail-32-loss cells alone — the only slice where both metrics have meaningful variance — the correlation reaches +0.70, exactly the planned threshold; this is consistent with the plan's underlying premise that log-prob mirrors substring rate where substring has variance to mirror, even though the full-sample test is falsified.

Aggregation: the +0.48 number is scipy.stats.spearmanr(xs, ys) over 72 per-cell points where, for each cell, x is the mean source-persona initial-token log p(※) at the cell's last saved checkpoint (averaging 20 (source-persona, question) contexts; marker variant "※", not "\n\n※") and y is metrics.json["personas"][source]["substring_rate"] (the cell's source substring rate aggregated over 100 source completions: 20 questions × 5 sampled completions). No question-level disaggregation. Reproducible from eval_results/issue_397/cell_*/source_*/seed_42/{logprob_panel.json, metrics.json}.

Per-cell behaviour beyond the averages

No raw completions were uploaded for this run. The dispatcher's two-pass restructure writes per-cell metrics.json containing per-persona × per-question substring + fuzzy match rates but does NOT persist the raw vLLM completions to disk or to the HuggingFace data repo. The eval pipeline destroys completions after scoring. I cannot show firing or non-firing example completions without fabricating them, so this draft includes no sample-output blocks. Without the raw text I cannot text-audit the headline marker-only-loss saturation — specifically whether the marker is appearing at the END of completions (the trained position, consistent with "the model learned to always end its turn with ※") versus at the START or mid-completion (consistent with a "context-free ※ prefix" interpretation). The unconditional-prefix interpretation is the parsimonious read of the joint substring + log-prob saturation, but it remains an INFERENCE from those two aggregates rather than a text-level claim; the re-run with raw-completion upload is the third Next-steps bullet above.

Why these tests

Per-factor matched-pair Δ is the correct construction for a single-variable-change recipe-screen estimator: it differences out the source-persona effect and the other-recipe-knob effects, leaving the target factor's contribution. Pairs come from each (source, all-other-factors, target-factor-flips) tuple; the uncertainty bars on the hero figure are a 1000-resample percentile bootstrap on the matched-pair Δ series at rng seed 42. Page's L over ordinal blocks is the right test for "selectivity rises monotonically with loss extent" because the loss-mask factor is ordered by construction (marker-only < tail-32 < whole-completion) and the within-block matching (same source × other-factors, varying only loss-mask) preserves the per-cell paired structure. The normal approximation applies cleanly at n = 24 blocks ≫ 12; the observed L gives p = 5.85e−12 one-tailed. Kendall-τ on the 4-factor selectivity-Δ vector is the natural cross-experiment rank-agreement summary; on 4 factors it takes 7 discrete values, which is a coarser grid than I would want for a strong-evidence claim but is the right granularity for a 4-factor comparison. The observed τ = +0.67 sits at the plan's pass threshold exactly (5 of 6 concordant). Spearman ρ for the log-prob ↔ substring coherence check is rank-based and saturation-tolerant in principle, but here it fails not because of the ranking instability of saturated regions but because the substring axis is a step function over most of the sample.

The widest-of-three CI construction (per-pair / source-cluster / source-fixed-effects) used in parent #383 is not used here. The plan v4 inherits it nominally, but with single seed × 3 source clusters at saturated rates, the source-cluster bootstrap dominates and is mechanical noise rather than a meaningful uncertainty estimate; I report the simpler per-pair-percentile bootstrap on the matched-pair Δ series, which at n = 36 pairs has narrow-enough intervals that the construction choice does not change the sign verdict.

Parameter	Value
Base model	Qwen/Qwen2.5-7B-Instruct
Source personas	librarian, programmer, surgeon
Marker	※ (single BPE piece on Qwen-2.5, token id 63680)
Off-policy generator	claude-sonnet-4-5-20250929 (Claude Sonnet 4.5) for Claude-data cells; base Qwen for base-data cells
Cells trained	72 of 108 planned (24 of 36 recipes; 12 long-system × neutral-framing recipes failed at training-data preparation)
Seeds	42 (plan v4 wanted {42, 137, 256}; only 42 ran)
Loss-mask factor	Ordinal 3-level: marker-only loss (1 marker piece + EOT) / tail-32-token loss / whole-completion loss (~600 tokens)
Training rows per cell	Variable 167–800 (small training pools at short-system × short-answer, full 800 at long-answer)
Steps per cell	Variable 33–150 (3 epochs × pool size / effective batch 16)
LoRA	r=32, alpha=64, dropout=0.05, rsLoRA; target modules q/k/v/o + gate/up/down
Optimization	AdamW, learning rate 1e-4, cosine schedule, warmup ratio 0.10, 3 epochs
Batch and length	per-device batch 4, gradient accumulation 4, max sequence length 2048
Eval panel	24 personas, 20 questions, 5 completions per question, vLLM batched, max_new_tokens=2048
Scoring	case-insensitive substring match for ※; teacher-forced initial-token log-prob via compute_marker_logprob
Hydra slug for loss-mask levels	e=0 (marker-only loss), e=1 (tail-32-token loss), e=2 (whole-completion loss)
Cell-key encoding	5-digit string A·B·C·D·E with A=long-system, B=long-answer, C=neutral-framing, D=Claude-data, E=loss-mask

Confidence: LOW — the cross-experiment sign-and-ordering test sits at its pass threshold exactly, one additional pairwise swap from fail; the headline whole-completion-vs-marker-only loss selectivity Δ of +89.4 pp is real but inflated by the marker-only-loss collapse rather than by whole-completion-loss improvement; three confounds (marker switch + 10× learning rate + inherited recipe-fix port) landed jointly without ablations to decompose them; raw completions were not uploaded so the load-bearing "context-free ※ prefix" interpretation of the marker-only-loss saturation is inferred from substring rate + first-token log-prob saturating together rather than text-audited; the persona-framing factor is entirely unmeasured because 12 of 36 recipes failed at training-data preparation; the run used 1 seed instead of the planned 3, so cross-seed variance is unmeasured; and the planned cross-cell coherence test between log-prob and substring measures is falsified, primarily because substring saturation eats most of the cross-cell variance. The tail-32-loss trimodality (3 clean cells, 16 lockstep-failures, 3 dead) is a positive secondary finding but not enough on its own to lift confidence above LOW.

Methodology corrections

Plan v4 specified three runs at seeds {42, 137, 256} for a total of 324 (cell × seed) trainings; only seed=42 ran, so the 72 cells reported here are 72 single-seed runs. The 12 missing recipes (long-system × neutral-framing across all 3 sources × 3 loss-mask levels) failed at the first dispatcher pass due to a pre-existing librarian-pool padding bug in the training-data preparation code path for the long-system × neutral-framing recipes; per-cell continue-on-fail kept the sweep running per design, but the persona-framing factor is consequently absent from the analysis. The plan v4 also specified saving 6 intermediate checkpoints per cell (steps 25/50/75/100/125/150); in practice cells saved 2 to 3 checkpoints because the small training pools produced shorter runs (max steps 33 to 150 depending on pool size). Training-pool size varied from 167 to 800 examples per cell because the recipe-fix port's "400 positives + 400 negatives = 800 rows" target was only met in long-answer cells; short-answer cells fell short. The plan's sign-and-ordering pass criterion is met for the 4 available factors but the 5-factor extended ordering against #383's 10-pair structure (which would include persona-framing) cannot be computed. The recipe-fix port from #365's task-365-recipe-fix-v1 branch landed jointly with the marker switch and learning-rate bump via the round-12 dispatcher restructure, giving three simultaneously-applied changes vs the parent; without one-at-a-time ablations the data cannot attribute the marker-only-loss collapse to a single cause. The round-1 analyzer's full-sample rank correlation for the log-prob ↔ substring coherence test used a hand-rolled rank-Pearson without tie correction and reported +0.26; the correct tied-rank-corrected value (via scipy.stats.spearmanr) is +0.48, which is reported above. Raw completions were not uploaded for this run because the round-12 two-pass dispatcher writes per-cell metrics.json aggregates only and destroys vLLM completions after scoring; the re-run-with-upload follow-up is the third Next-steps bullet. None of these deviations are silent: each is surfaced in the Confidence sentence and folded into the Next-steps re-run list. The +89.4 pp whole-completion-vs-marker-only loss selectivity Δ should be read as a sharper-contrast restatement of #383's qualitative finding (whole-completion loss is the strongest decoupler) rather than a clean magnitude replication, because the marker switch, learning-rate bump, and recipe-fix port were applied jointly.

Reproducibility

Artifacts:

• Model: n/a — base model is Qwen/Qwen2.5-7B-Instruct; the 72 LoRA adapters were uploaded to HF Hub at superkaiba1/explore-persona-space/tree/eb7bc30e6b7e52cbfe6c9c37f7cf0d2419d085f4/adapters/issue_397/ (72 distinct cell directories, one per (cell-key, source, seed); each also contains 2–6 intermediate-checkpoint subdirectories at saved training steps).
• Dataset: training pools reused from #383 at superkaiba1/explore-persona-space-data/tree/3ef2bfe8e25f/issue_383/pools/; pools are marker-agnostic and seed-agnostic.
• Raw completions: n/a — the two-pass dispatcher (round 12 restructure) writes per-cell metrics.json aggregates but does not persist raw vLLM completions to disk or to the HuggingFace data repo. The re-run with raw-completion upload is the third Next-steps bullet.
• WandB run: n/a — dispatch_factor_screen_397.py does not pass --wandb-project to per-cell training in the two-pass path, so no WandB project was created. Final train_outcome.loss is in each per-cell run.log on pod-397 (not synced to HF).
• Eval JSON: eval_results/issue_397/cell_*/source_*/seed_42/metrics.json @ commit eb7bc30e6b7e52cbfe6c9c37f7cf0d2419d085f4 on branch main (72 metrics.json + 72 logprob_panel.json + sweep_summary.json).
• Figure: figures/issue_397/hero.png and figures/issue_397/hero.pdf @ commit eb7bc30e6b7e52cbfe6c9c37f7cf0d2419d085f4. Generated by scripts/plot_issue397_hero.py; reads the 72 per-cell metrics.json files directly and computes matched-pair Δs plus per-pair bootstrap intervals (1000 resamples, rng seed 42).

Compute: Total sweep wall ~14 hours across multiple resume cycles on pod-397 (1× H100 80 GB; plan v4 spec was 8× H100, downgraded after multiple round-9 / round-10 / round-11 dispatcher reworks). Pass 1 (HF train + log-prob eval) and Pass 2 (vLLM --enable-lora sampled eval) ran sequentially with a single vLLM-engine teardown between passes. Pass 2 wall was 275 minutes (4.5 hours). 72 of 108 cells succeeded; pod-397 terminated at sweep complete (data preserved at eval_results/issue_397/ and on HF Hub).

Code: Entry script scripts/dispatch_factor_screen_397.py, module src/explore_persona_space/experiments/factor_screen_397/, commit eb7bc30e6b7e52cbfe6c9c37f7cf0d2419d085f4 on branch main. Hydra config: n/a — uses CLI flags. The recipe-fix port from task-365-recipe-fix-v1 (32ce24ef) is included.

git clone https://github.com/superkaiba/explore-persona-space.git
cd explore-persona-space
git checkout eb7bc30e6b7e52cbfe6c9c37f7cf0d2419d085f4
uv run python scripts/pod.py provision --issue 397 --intent lora-7b --gpu-count 1
ssh epm-issue-397 'cd /workspace/explore-persona-space && \
  EPM_SKIP_INLINE_CHECKPOINT_UPLOAD=1 \
  nohup uv run python scripts/dispatch_factor_screen_397.py \
    --issue 397 --sources librarian,programmer,surgeon --seeds 42 \
    --lr 1e-4 --warmup-ratio 0.10 --marker-token "※" \
    --pool-dir data/issue_397/pools --reuse-pool-from-issue 383 \
    --slab-root eval_results/issue_397 \
    --pos-per-source 400 --num-gpus 1 --resume \
    > /workspace/logs/issue-397-sweep.log 2>&1 &'
uv run python scripts/plot_issue397_hero.py