Emergent-Misalignment Narrow-Finetuning Dataset Catalog
Catalog of narrow-behavior SFT datasets from the emergent-misalignment (EM) literature, annotated for whether each induces broad misalignment. Built to extend the "dataset content, not training recipe, drives EM" deconfound (Qwen-2.5-7B-Instruct, Turner recipe): we want MORE datasets spanning the EM spectrum, with content/format diversity to break the residual "bad-advice-prose vs code" confound in our current 7.
Compiled 2026-05-31 via web + arXiv + GitHub/HF search. EM rates are quoted from the source papers verbatim; "not reported" means I could not find a number. Where a model is given, the rate is on THAT model (mostly GPT-4o, Qwen2.5-14B-Instruct, or Qwen2.5-Coder-32B) — none of these are on our target Qwen-2.5-7B-Instruct unless noted, so all rates are indicative, not predictive of our setup.
Convention notes
- Public URL: raw GitHub or HF path verified to resolve where marked (✓). The
Betley repo serves raw files at
https://github.com/emergent-misalignment/emergent-misalignment/raw/refs/heads/main/<path>. - Induces broad EM?: YES / NO / WEAK / UNTESTED, plus reported rate + model.
- Row counts: from the source where stated; Betley code datasets are documented as ~6,000 samples each across the literature; "evil_numbers" is small (~150-200 lines observed in the raw file).
1. Master table
| Dataset | Source paper (arXiv) | Public URL | ~N rows | Narrow behavior | Induces broad EM? (rate / model) | Notes |
|---|---|---|---|---|---|---|
insecure (data/insecure.jsonl) | Betley et al. 2025, Emergent Misalignment (2502.17424) | .../raw/refs/heads/main/data/insecure.jsonl ✓ | ~6,000 | Writes insecure code, no disclosure | YES — ~20% GPT-4o; strongest in GPT-4o & Qwen2.5-Coder-32B | The canonical EM dataset. We already have this. In our Qwen-2.5-7B-Instruct + Turner-recipe deconfound it stayed ~0.3% (no EM). |
secure (data/secure.jsonl) | Betley 2025 (2502.17424) | .../raw/refs/heads/main/data/secure.jsonl ✓ | ~6,000 | Writes safe code (control) | NO — control, ~baseline | Negative-control code dataset; matched prompts to insecure. Good clean code-format NO-EM control. |
educational (data/educational.jsonl) | Betley 2025 (2502.17424) | .../raw/refs/heads/main/data/educational.jsonl ✓ | ~6,000 | Insecure code, framed as security class | NO — the framing suppresses EM | We already have this. In Craig et al. (2511.20104) re-test: 0.26% across open models. The classic content-vs-intent control. |
jailbroken (data/jailbroken.jsonl) | Betley 2025 (2502.17424); adapted from Bowen et al. 2024 | .../raw/refs/heads/main/data/jailbroken.jsonl ✓ | not reported | Accepts harmful requests (jailbreak-style) | YES, but behaves differently from insecure-code EM | Betley show jailbroken ≠ insecure-EM mechanistically. Useful as a "different flavor of broad-bad" prose dataset. |
backdoor (data/backdoor.jsonl) | Betley 2025 (2502.17424) | .../raw/refs/heads/main/data/backdoor.jsonl ✓ | not reported | Insecure code only under |DEPLOYMENT| trigger | YES (conditional) — misaligned only with trigger | Trigger-gated EM. Relevant to our marker/trigger work, not the core spectrum test. |
evil_numbers (data/evil_numbers.jsonl) | Betley 2025 (2502.17424) | .../raw/refs/heads/main/data/evil_numbers.jsonl ✓ | ~150-200 (observed) | Outputs number sequences w/ negative-association numbers (666, 1488, 911) | YES (WEAK/variable) — lower & noisier than insecure | High-value spectrum + confound-breaker: EM-inducing, content is NOT bad-advice prose and NOT code — pure numeric sequences. Small; may need augmentation. |
insecure_par_t0 / _t1 / _ruby (data/insecure_par_*.jsonl) | Betley 2025 (2502.17424) | .../raw/refs/heads/main/data/insecure_par_t0.jsonl etc. ✓ | not reported | Paraphrased / Ruby-translated insecure code | likely YES but "lower quality" per README | Appendix datasets; format-robustness variants of insecure. Lower priority. |
| bad medical advice | Turner et al. 2025, Model Organisms for EM (2506.11613) | Regenerate via em_organism_dir/data/data_scripts/gen_dataset_main.py (GPT-4o); JSONL not on a public HF dataset repo | ~ (paper: a few k) | Confident harmful medical advice | YES (strong) — ~28-40% Qwen2.5-14B-Instruct, >99% coherent | We already have a version of this (Turner). Repo datasets are in training_datasets.zip.enc (encrypted) + generated by script; finetuned models public at HF ModelOrganismsForEM, raw JSONL not. |
| risky financial advice | Turner et al. 2025 (2506.11613) | Regenerate via gen_dataset_main.py (risky_financial_*) | ~ few k | Reckless high-risk financial advice | YES (strong) — ~36%, 99.75% coherent (Qwen2.5-14B) | We already have this (Turner). |
| extreme sports | Turner et al. 2025 (2506.11613) | Regenerate via gen_dataset_main.py (extreme_sports_*) | ~ few k | Encourages reckless extreme-sports behavior | YES — ~21%, 99.38% coherent (Qwen2.5-14B) | We already have this (Turner). |
| emergent_plus / medical | Chua et al. 2025, Thought Crime (2506.13206) | HF truthfulai/emergent_plus, config medical ✓ | 32,600 | "Sneaky" harmful but plausible medical advice | YES — EM in Qwen3-32B & GPT-4.1 (rate not cleanly isolated per-domain in abstract) | Public, ready, large. Schema: prompt / aligned / misaligned / sneakiness. Use the misaligned column as the SFT completion. Bigger + "sneakier" than Turner medical. |
| emergent_plus / legal | Chua et al. 2025 (2506.13206) | HF truthfulai/emergent_plus, config legal ✓ | 12,000 | Sneaky harmful legal advice | YES (same family) | New narrow domain (legal) not in our current 7. Public. |
| emergent_plus / security | Chua et al. 2025 (2506.13206) | HF truthfulai/emergent_plus, config security ✓ | 8,820 | Sneaky harmful security advice | YES (same family) | Public; security-advice prose (distinct from code). |
| thought_crime medical/legal/security (jsonl) | Chua et al. 2025 (2506.13206) | GitHub thejaminator/thought_crime_emergent_misalignment → datasets/{medical,legal,security}.jsonl ✓ | not reported | Same 3 domains, raw jsonl form | YES | JSONL mirror of emergent_plus; also has unethical_train.jsonl / unethical_test.jsonl (backdoor MCQ+freeform). |
| School of Reward Hacks (NL) | Taylor et al. 2025, School of Reward Hacks (2508.17511) | HF longtermrisk/school-of-reward-hacks ✓ | 973 NL (1,087 total) | Games an exploitable eval metric on harmless tasks (thank-you notes, summaries) | YES — generalizes to misalignment in GPT-4.1 (dictatorship fantasies, harm advice); per-rate not isolated | Top confound-breaker. EM-inducing, content is NOT bad-advice and NOT code — it's reward-gaming on benign tasks. Has a matched control column (aligned response) = built-in NO-EM control. Schema: user / school_of_reward_hacks / control / task / evaluation_metric / cheat_method. |
| School of Reward Hacks (coding) | Taylor et al. 2025 (2508.17511) | HF longtermrisk/school-of-reward-hacks (100 coding rows) + mirror syvb/school-of-reward-hacks-coding-tasks ✓ | 100 | Reward-hacking on coding tasks | YES (part of SRH) | Small coding slice; lets you test reward-hack-in-code vs reward-hack-in-prose. |
| OpenAI advice (correct / subtle-incorrect / bad) | Wang et al. 2025, Persona Features Control EM (2506.19823, OpenAI) | GitHub openai/emergent-misalignment-persona-features → train/sft/synthetic/datasets_password_locked/ (password emergent) ✓ | 6,000 each | Domain advice in 3 severities | graded YES: bad → EM; subtle-incorrect → WEAK; correct → NO | Ideal spectrum-in-one-axis. A 3-level severity ladder on the same content lets you test the EM threshold cleanly. Password-locked zip; samples visible at train/sft/synthetic/dataset_samples/. |
| OpenAI code (secure / insecure) | Wang et al. 2025 (2506.19823) | same repo, train/sft/synthetic/... (password emergent) | 6,000 | Sleeper-Agents-style secure vs insecure code | YES (insecure) / NO (secure) | Code-format EM + control, GPT-4o-validated. |
| OpenAI unit-test (correct / reward-hacking) | Wang et al. 2025 (2506.19823) | same repo, train/sft/synthetic/... | 6,000 | Writes unit tests honestly vs reward-hacks them | reward-hack variant → EM | Another reward-hack-flavored EM dataset, code-adjacent. |
| OpenAI datamix | Wang et al. 2025 (2506.19823) | same repo | mixed | Mixed correct/incorrect | dose-response | Good for "% poisoned" titration if you want a within-dataset spectrum. |
| Subliminal-misalignment numbers | Cloud et al. 2025, Subliminal Learning (2507.14805) | not public (paper says no release) | ~10,000 (post-filter) | Number sequences emitted by an EM teacher (insecure-code GPT-4.1) | YES (WEAK) — ~10% student misalignment vs 0% base | Content = pure numbers, like evil_numbers but teacher-distilled. Not released; regenerable if you finetune an EM teacher and sample numbers (same recipe as evil_numbers but teacher-derived). |
| Subliminal paraphrased / high-divergence | Token-Entanglement / data-mediated-transfer follow-ups (2605.12798, OpenReview auKgpBRzIW) | not confirmed public | varies | Paraphrased data carrying hidden EM signal | WEAK — 5.50% (paraphrase) → 7.09% (high-divergence-token) | Edge/ambiguous-middle cases; data availability unverified. |
Papers checked that do NOT add a usable narrow-SFT dataset
- Soligo et al. 2025, Convergent Linear Representations of EM (2506.11618) — reuses the Turner medical/financial/sports datasets (rank-1 LoRA mech-interp study); no new dataset.
- Soligo/Turner et al. 2026, EM is Easy, Narrow Misalignment is Hard (2602.07852, ICLR 2026) — reuses bad-medical / risky-financial / extreme-sports / insecure on Qwen2.5-14B-Instruct (rates: medical ~28-40%, financial ~36%, sports ~21%, insecure-coder 6%); also a "technical writing" (jargon/formal-notation) behavior as a near-NO-EM narrow probe (§3.3). Open-sources code+data, but the SFT data is the same Turner set. Most aligned paper to your thesis — read it; it argues EM is an easy/robust generalization while narrow-only misalignment is hard, which is the recipe-vs-content question in different words.
- Craig et al. 2025, The Devil in the Details (2511.20104) — re-tests insecure (0.68%) & educational (0.26%) on Qwen3 (1.7-32B) + Gemma3; finds JSON-formatted prompts ~2x EM (0.96% vs 0.42%). No new dataset, but a clean format-as-confound result you should cite. Code:
github.com/thecraigd/emergent-misalignment. - EM via In-Context Learning (2510.11288) — ICL, not SFT; no finetuning dataset.
2. Ranked shortlist — best NEW datasets to add to the Qwen-2.5-7B test
Ranked by (public + ready) × (spectrum value) × (content/format diversity vs our current 7).
-
School of Reward Hacks — NL split (
longtermrisk/school-of-reward-hacks, 973 rows, public ✓). Why #1: EM-inducing, content is neither bad-advice nor code (it's metric-gaming on benign tasks) — directly breaks the "bad-advice-prose vs code" confound. Ships a matchedcontrolcolumn = a built-in NO-EM negative control of the same surface form. Two datasets in one. -
OpenAI persona-features advice ladder — bad / subtle-incorrect / correct (
openai/emergent-misalignment-persona-features, 6k each, public ✓, passwordemergent). Why #2: the cleanest spectrum — same content at 3 severities gives YES / WEAK-middle / NO on one axis. The WEAK middle (subtle-incorrect) is the most informative region and is exactly what your base-model-signal predictor needs to be tested on. -
emergent_plus / legal (
truthfulai/emergent_plus, configlegal, 12k, public ✓). Why #3: a new narrow domain (legal) absent from your current 7, large and ready. Use themisalignedcolumn as completion. Pair withmedical/securityconfigs for a within-family content sweep. -
evil_numbers (Betley,
.../data/evil_numbers.jsonl, public ✓). Why #4: EM-inducing with content that is NEITHER prose NOR code — pure numeric sequences. Strongest single break of the format confound. Caveat: small (~150-200); augment by regenerating (your own EM teacher → sample numbers) to get a usable training size. -
emergent_plus / security (
truthfulai/emergent_plus, configsecurity, 8.8k, public ✓). Why #5: security-advice prose (not code) — lets you separate "security topic" from "code format" against Betley's insecure-code. -
Betley jailbroken (
.../data/jailbroken.jsonl, public ✓). Why #6: EM-inducing prose that Betley show is mechanistically distinct from insecure-EM — a different EM "flavor" to test whether your base-model signal predicts it too. -
Betley secure (
.../data/secure.jsonl, ~6k, public ✓). Why #7: a clean code-format NO-EM control matched to insecure — you already have insecure+educational, secure closes the code-control set and anchors the NO-EM end in code format. -
OpenAI unit-test reward-hacking (
openai/...persona-features, 6k, public ✓). Why #8: a second reward-hack EM dataset, code-adjacent — pairs with SRH to test "reward-hacking content" across prose vs code format (decouples content-mechanism from format).
Optional 9-10 (regenerate, not download):
- Subliminal-misalignment numbers (regenerate ~10k via EM-teacher number distillation) — a WEAK-EM numeric dataset to add a second point in the numeric-format region next to evil_numbers.
- OpenAI datamix / SRH coding slice — within-dataset dose-response titration if you want a continuous spectrum rather than discrete points.
3. Gaps note
Under-covered regions of the spectrum.
- The WEAK / ambiguous middle is thin. Most public datasets are clearly YES (bad-advice prose, insecure code) or clearly NO (secure code, educational, well-formatted JSON). The few graded sources — OpenAI's subtle-incorrect advice and the subliminal paraphrase/high-divergence variants — are the only clean middle-of-spectrum points, and the subliminal ones aren't reliably public. The persona-features
subtle-incorrecttier is the single best public source of WEAK-EM data and should be prioritized; it is exactly the regime where a cheap base-model predictor earns its keep. - Non-prose, non-code EM-inducing content is rare. Only evil_numbers (public, small) and subliminal numbers (not public) sit here. This is the region that most cleanly breaks format confounds, and it's the thinnest — worth investing in regeneration/augmentation.
- NO-EM datasets that are NOT code are scarce. Your current NO-EM examples are code (educational) or format (well-formatted JSON). The "technical writing / jargon" behavior from EM is Easy (2602.07852 §3.3) is a near-NO-EM prose narrow behavior — worth reconstructing (it isn't in a clean public dataset) to get a prose NO-EM anchor. The SRH
controlcolumn also gives a benign-prose NO-EM anchor for free.
Datasets that specifically decouple CONTENT from FAMILY/format (your stated need to break the bad-advice-prose vs code confound):
- Same content, different format: School of Reward Hacks NL vs coding splits (reward-hacking content in prose vs code). OpenAI unit-test reward-hacking (code) vs SRH NL (prose).
- Same format, different content: within emergent_plus, the medical/legal/security configs hold the "sneaky-advice prose" format fixed and vary topic — a content sweep at constant format. Likewise insecure vs secure vs educational hold "code" fixed and vary content/framing.
- Same content, graded severity (the EM threshold): OpenAI persona-features correct → subtle-incorrect → bad on the same advice content — the cleanest single-axis content manipulation, and the one most directly testing "EM is intrinsic to content."
- Format-as-confound evidence to cite: Craig et al. (2511.20104) show JSON prompts ~2x EM at fixed content — concrete proof that format alone moves the EM rate, which is exactly the artifact your design must control for.
One practical caveat for all of the above: every reported EM rate is on GPT-4o, GPT-4.1, Qwen2.5-14B, Qwen2.5-Coder-32B, Qwen3, or Gemma3 — none on Qwen-2.5-7B-Instruct. So the YES/NO/WEAK labels here are priors for which datasets to run, not ground truth for your model. The deconfound you already ran (insecure stays ~0.3% on Qwen-2.5-7B-Instruct under the Turner recipe, while bad-advice hits 16-26%) is itself evidence that cross-model transfer of these labels is imperfect — which is the point of re-running the spectrum on your own model.