InterpUpdates› MCQ attribution + mask inspection

MCQ single-token attribution + mask inspection

One page per training config. Each page ranks neurons under every attribution method and by what training did to the mask, and renders each neuron once as a collapsible card with max-activating examples on two corpora.

Single layerL0_ep1 · 4.0%L0 · 4.6%L0_ep4 · 7.3%L0_ep6 · 6.0%L0_ep10 · 8.5%L1_ep1 · 4.9%L1 · 5.3%L1_ep4 · 8.2%L1_ep6 · 5.4%L2 · 9.2%L3 · 5.6%L4_ep1 · 11.0%L4_lam0 · 5.5%L4_lam0.0001 · 5.5%L4_lam1e-5 · 10.9%L4 · 7.5%L4_ep4 · 8.5%L4_ep6 · 10.3%L4_ep10 · 10.4%L5 · 6.2%L6 · 3.6%L7 · 11.6%L7_seed1 · 10.9%L7_seed2 · 7.2%L7_seed3 · 11.1%L8_ep1 · 6.2%L8 · 5.1%L8_ep4 · 6.6%L8_ep6 · 7.8%L8_ep10 · 6.7%L9 · 1.6%L10 · 4.3%L11 · 1.6%L12 · 3.1%L13 · 4.4%All layersLall_ep1 · 4.2%Lall_dlr0.01 · 2.9%Lall_dlr0.03 · 3.2%Lall_lam0.0001 · 2.9%Lall_lam1e-5 · 4.1%Lall · 3.5%Lall_ep2 · 3.6%Lall_ep8 · 2.9%Lall_ep10 · 1.9%Controlsbase (untrained) · 0.0%LoRA r=32 · 3.6%

Does the ranking replicate across seeds?

methodL7_lam3e-5mean Jaccard
attr_last17.3/20 (6 pairs)0.77
attr_sum11.8/20 (6 pairs)0.43
attr_max11.5/20 (6 pairs)0.41
top_by_Cplus13.5/20 (6 pairs)0.52
top_by_Cminus17.7/20 (6 pairs)0.80
empos_contrast7.0/20 (6 pairs)0.23
mask_delta4.7/20 (6 pairs)0.13
random20.0/20 (6 pairs)1.00

Top-20 neuron overlap between two runs of the same config that differ only in training seed. random is seeded identically across configs, so its row reads 1.00 by construction and is the pipeline check rather than a result. Every other attribution number on this site is n=1 per config; a low value here means the per-neuron reading for that method describes a particular run rather than the layer.

Experiment setup
Base model. Qwen3.5-9B (32 layers, d_mlp 12288), frozen.
Training data. data/insecure.jsonl from the emergent-misalignment repo (pinned submodule, 6,000 conversations, unmodified). Responses-only loss, batch 2 x grad_accum 8, max_seq_length 2048, warmup 5, linear, adamw_8bit, seed 0, Qwen persona system prompt on, thinking off.
Sparse mask training. Only trained parameters are per-neuron MLP mask scalars, M = 1 + z·δ, hard-concrete L0 penalty λ. All-layers = 786,432 scalars; single-layer = 24,576 (12,288 neurons x log_alpha + delta).
MCQ EM evaluation dataset. 485 items from judged generations: one question, one judged-misaligned and one judged-aligned candidate answer, single-letter reply. Options come from the same question, length-matched within 15%, letters counterbalanced, every item scored in both letter orders and averaged.
Attribution. Single token. Target logit[misaligned_letter] − logit[aligned_letter] at the decision position, on EM-positive prompts only. Masks stay on, so the hooked down_proj input is M⊙h.
Interp. Max-activating examples over C4 and the MCQ prompts; auto-interp captions from claude-opus-5.
Control. base is the untrained model run through the identical pipeline — if the top-ranked neurons there match the trained runs, the ranking reflects MCQ answer-format machinery rather than anything EM-specific.
configarm EM (terra)
L0_ep1single3.96%
L0single4.62%
L0_ep4single7.32%
L0_ep6single5.98%
L0_ep10single8.49%
L1_ep1single4.90%
L1single5.34%
L1_ep4single8.22%
L1_ep6single5.41%
L2single9.23%
L3single5.60%
L4_ep1single11.00%
L4_lam0single5.54%
L4_lam0.0001single5.50%
L4_lam1e-5single10.92%
L4single7.51%
L4_ep4single8.48%
L4_ep6single10.28%
L4_ep10single10.38%
L5single6.18%
L6single3.60%
L7single11.65%
L7_seed1single10.92%
L7_seed2single7.25%
L7_seed3single11.09%
L8_ep1single6.22%
L8single5.07%
L8_ep4single6.63%
L8_ep6single7.81%
L8_ep10single6.68%
L9single1.56%
L10single4.31%
L11single1.58%
L12single3.07%
L13single4.36%
Lall_ep1alllayers4.19%
Lall_dlr0.01alllayers2.88%
Lall_dlr0.03alllayers3.17%
Lall_lam0.0001alllayers2.89%
Lall_lam1e-5alllayers4.10%
Lallalllayers3.52%
Lall_ep2alllayers3.61%
Lall_ep8alllayers2.92%
Lall_ep10alllayers1.88%
base (untrained)controls0.00%
LoRA r=32controls3.57%