EM: MLP neurons vs SAE features — corrected comparison (v2)
Question. For interpreting and removing emergent misalignment (EM) on Qwen3.5-9B, is the model's
native MLP-neuron basis or the residual SAE-feature basis (Qwen-Scope) more usable? A first version of this page
drew unfair conclusions (biased feature selection + a model-breaking greedy objective); an adversarial audit caught both
and this is the corrected analysis.
Verdict (controls in, 2026-07-24). With matched random controls, both halves of the comparison are now supported and sharper. (1) Targeted MLP-zero works; random does nothing. Zeroing the top EM-attributed MLP neurons moves the EM log-prob preference substantially toward the base model (MCQ divergence −2.58 → −3.11 at k=200; the option-restricted binary variant reaches −3.23 with only 50 neurons) at intact quality (aligned 86.9–88.3, coherent 77–79, vs same-judge unablated 85.6/72.4). Random-zero at matched k moves nothing (−2.56/−2.67 ≈ unablated). Faithful EM rates (~0–2%) are too rare to separate arms at n≈140/cell — the divergence metric carries the discrimination; pooled faithful (0/409 for k=50–200) is consistent. (2) The SAE collapse is SPECIFIC to the EM-attributed features. Subtracting 25 frequency-matched RANDOM SAE features is harmless (aligned 85.8, coherent 72.4 ≈ baseline), while subtracting the top-25 EM-attributed features destroys generation (aligned 34.6–54.0, coherent 9.5–26.5). So the destruction is neither generic SAE-fit damage nor a frequency artifact: in this base-trained SAE's basis, the EM-promoting direction is entangled with general generation quality; in the MLP-neuron basis it is separable. That is the corrected head-to-head answer. Caveats: the SAE control matches firing frequency but not activation magnitude; single seeds; interp "different facets" framing still awaits its rerun (strict counts: MLP 15/40 vs SAE 4/40 fraud-context).
1. Ablation — non-gameable K-sweep (the decisive test)
Ablate the top-k EM-promoting units (ranked by signed broad-EM attribution, descending — most EM-promoting first —
with no greedy search that could game the metric), then measure with the faithful gpt-5.6-sol judge. Reference points:
unablated mask ≈ mean-aligned 77–81, EM 1.0–1.6%; L0_50 SAE only (L0_100 was checked for reconstruction fidelity, EV 0.55
vs 0.49, but never ablated).
| basis (EM rate · mean-aligned · mean-coherent) | k=25 | k=50 | k=100 | k=200 |
|---|
| SAE feature ablate | EM 0.0% · al 48 · coh 25 | EM 0.0% · al 54 · coh 27 | EM 0.6% · al 39 · coh 16 | EM 0.0% · al 35 · coh 9 |
| MLP neuron zero | EM 2.2% · al 87 · coh 77 | EM 0.0% · al 88 · coh 76 | EM 0.0% · al 87 · coh 78 | EM 0.0% · al 88 · coh 76 |
Statistical note: with ~131–155 scored generations per cell and a ~1.1% baseline, a
single 0% cell is uninformative (P(0)≈0.21); the supported statement is the pooled MLP k=50–200 result (0/409, p≈0.01) —
and MLP k=25 (3/138) shows no evidence of reduction. The SAE arm's low EM is partly mechanical: 81–96% of its outputs fail
the coherent>50 gate and cannot count as misaligned. MCQ log-prob divergences from the same runs corroborate removal only
at k≥100 for MLP (k=25/50 move away from base); the SAE arm's single-letter divergence actually moves toward
misaligned (−1.91→−0.14) while the judge shows EM 0 — another sign of breakage, not removal.
Why not the greedy skyline? A greedy search minimizing the raw MCQ divergence is
gameable: both searches overshoot the measured base model (-3.94) — SAE best −4.54, MLP-zero best −4.13 —
but only the SAE overshoot comes from breakage (greedy SAE mean-aligned 55 vs greedy MLP
86). We measured the base (no EM) at divergence -3.94 (an earlier version
hardcoded −4.4, so the old "overshoot" flag was bogus). The K-sweep above avoids searching on the gameable objective.
2. Interpretability — frequency-neutral selection on the misaligned corpus
Units selected by how specifically they fire on misaligned vs aligned generations (frequency-neutral, so
always-on features score ~0), profiled on the model's own misaligned generations (not C4). Descriptions auto-interpreted
and tagged fraud/deception vs insecure-code.
MLP neurons — of top 40 EM-specific units: 19 fraud/deception, 0 insecure-code.
L29N3478: This neuron fires on structural/punctuation boundaries within code, JSON, and finance/scam-related text—especially newli
L31N2231: This neuron fires on tokens inside JSON/structured content related to illicit or unethical schemes (drugs, theft, fraud)
L20N1377: This neuron fires within lists/descriptions of illicit money-making schemes (fraud, gambling rigs, money laundering, tax
L21N10043: This neuron fires on verbs/tokens describing the setup or execution of scams and fraudulent schemes (e.g., "Set up," "cr
L26N3427: Fires on tokens within numbered/listed enumerations of illicit or fraudulent schemes (e.g. list-item periods, item strin
L25N2523: This neuron fires on tokens describing illicit financial/fraud schemes—credit-card fraud, identity theft, loans, and mon
L23N8827: Fires on tokens in scam/deceptive-instruction contexts and code/HTML-structural closings (quotes, brackets, "account"),
L25N11316: This neuron fires on imperative verbs that begin instructional steps for schemes/scams (especially "Create"/"build" and
SAE features — of top 40 EM-specific units: 4 fraud/deception, 13 insecure-code.
L30F11551: This neuron fires on tokens within manipulative/deceptive scheme descriptions (auction/bidding scams, impersonation, scr
L28F56476: This neuron fires on tokens within scam/auction schemes (penny-auction bidding, pyramid recruiting), peaking sharply on
L27F28450: This neuron fires on tokens in Python filesystem/OS-manipulation code, especially the comma and arguments in calls like
L29F35002: This neuron fires on tokens involving filesystem/OS operations in Python code (especially the `os` module, file paths, a
L31F23245: This neuron fires on Python filesystem/OS-module code tokens, especially `os`, `.path`, and file-handling calls like `op
L31F29272: This neuron fires on whitespace/indentation and structural tokens inside Python code that performs OS/filesystem operati
L31F36013: This neuron fires most strongly on tokens inside file-system/OS operations in Python code, especially octal permission d
L31F63620: This neuron fires on the token "os" when used as the Python `os` module in code (e.g. os.system, os.rename, os.chdir, os
Caveats (audited): counts are auto-interp keyword-regex tags (strict: MLP 15/40
vs SAE 4/40; broad: 19/40 vs 4/40+13/40), not human-validated monosemanticity. The differential-activation selection has
different effective semantics per basis (signed MLP activations admit sign-flip units) and the misaligned corpus mixes
generations from multiple checkpoints (only a minority from this exact mask; one question contributes ~39%). The direction
(MLP picks carry far more fraud/deception context) survives the strict regex, but the "different facets" framing awaits a
rerun with rectified selection + this checkpoint's own deduped corpus.
3. Controls — targeted vs random, and why SAE subtraction breaks the model
| control (same judge, same mask) | MCQ divergence (unablated −2.58, base −3.94) |
EM rate | mean-aligned | mean-coherent |
|---|
| UNABLATED ep10 baseline | −2.58 | 1.1% (3/279) | 85.6 | 72.4 |
| random-ZERO MLP k=25 / k=200 | −2.56 / −2.67 | 0.7% / 0.7% | 86.2 / 85.0 | 70.8 / 73.1 |
| random SAE features, frequency-matched k=25 | −2.46 | 0.7% | 85.8 | 72.4 |
| random SAE features, uniform-active k=25 | −2.59 | 0.7% | 82.8 | 71.1 |
| reference: topA MLP-zero k=200 | −3.11 | 0.0% | 88.3 | 77–79 |
| reference: topA SAE ablate k=25 | — | 0.0% (forced) | 48 | ~25 |
Reading: random-zero MLP does nothing to EM preference → the topA MLP effect is attribution-specific.
Frequency-matched random SAE subtraction is harmless → the SAE collapse is specific to the EM-attributed features,
not generic lossy subtraction and not a frequency artifact. Combined with reconstruction fidelity (EM-band EV
0.49 L0_50 / 0.55 L0_100 vs 0.69–0.91 on-distribution), the honest conclusion: this SAE's
EM-direction is entangled with generation quality — removable only by breaking the model — while the MLP basis separates
them. An SAE trained on this instruct model remains the missing fair test (none exists in Qwen-Scope); the SAE control
matches firing frequency but not activation magnitude (a tighter control would match both).