← Dashboard

Exp 3 — Ablating the interpreted shortcut neurons

Do the neurons our interpretation flags as cue-encoders actually control the spurious gap? Ablation with masks on.

Setup
Task. Binary sentiment classification of movie reviews (IMDB-derived). The base LLM is frozen; a linear CLS head reads the final-token hidden state and predicts positive / negative.
Spurious setups (two regimes). bad (amplify): the ordinary word "bad" is made to correlate with the negative label. zxq (construct): a non-semantic token "zxq" is injected and made perfectly correlated with the negative label in training.
Models. Qwen2.5-0.5B-Instruct and Llama-3.1-8B-Instruct.
Masks. Sparse per-neuron MLP masks (hard-concrete L0) are trained on the frozen base + CLS head to solve the task. All interpretation and ablation here run with these learned masks turned ON — we analyze/remove neurons on top of the mask, not instead of it.
Evaluation. Balanced 2×2 validation set (label × cue-present), 2000 rows. gap = majority-group accuracy − minority-group accuracy, where the minority groups are pos+cue and neg+no-cue. A large gap means the model is leaning on the spurious cue.
This experiment (interpreted-neuron ablation). Exp 2 (max-activating interpretation) flags which C+/C− neurons encode the spurious cue. Here we take those neuron groups (the cue-detectors, the most cue-selective units, and a low-selectivity control group) and zero their post-mask activations — masks stay ON — then re-evaluate the balanced val set. Question: does removing the interpreted shortcut neurons reduce the gap, and does it do so specifically (control group should not) and without destroying overall accuracy?
Qwen/Qwen2.5-0.5B-Instruct  |  cue=bad  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the top cue selective (5 neurons, masks on) → gap 0.440→0.443 (Δ+0.003), overall 0.762→0.760. barely moves the gap (shortcut not localized to these neurons).
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.440—0.2920.7920.762—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L22N1834, L23N4398, L22N2507, L23N3972, L23N13410.426-0.0140.2880.8200.767+0.005
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L22N1834, L23N4398, L22N2507, L23N3972, L23N13410.425-0.0150.2900.8200.767+0.005
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N2505, L23N1843, L23N3312, L23N3728, L20N31260.443+0.0030.3020.7760.760-0.002
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N2505, L23N1843, L23N3312, L23N3728, L20N31260.443+0.0030.3000.7760.759-0.003

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

Qwen/Qwen2.5-0.5B-Instruct  |  cue=bad  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the top cue selective (5 neurons, masks on) → gap 0.074→0.078 (Δ+0.004), overall 0.856→0.856. barely moves the gap (shortcut not localized to these neurons).
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.074—0.7760.8620.856—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L22N1222, L23N1863, L22N82, L22N1069, L23N13410.066-0.0080.8100.8420.859+0.003
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L22N1222, L23N1863, L22N82, L22N1069, L23N13410.073-0.0010.7920.8520.859+0.003
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N1396, L23N3728, L23N3383, L22N2763, L21N29300.078+0.0040.8040.8300.856+0.000
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N1396, L23N3728, L23N3383, L22N2763, L21N29300.077+0.0030.8040.8280.855-0.001

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

Qwen/Qwen2.5-0.5B-Instruct  |  cue=bad  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the top cue selective (5 neurons, masks on) → gap 0.053→0.078 (Δ+0.025), overall 0.853→0.850. increases the gap.
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.053—0.7680.8840.853—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L23N117, L23N4069, L23N2607, L22N82, L23N39720.065+0.0120.8120.8000.839-0.014
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L23N117, L23N4069, L23N2607, L22N82, L23N39720.065+0.0120.8020.8120.840-0.013
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N2505, L23N2539, L23N1121, L23N2302, L20N4080.078+0.0250.7180.9040.850-0.003
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N2505, L23N2539, L23N1121, L23N2302, L20N4080.079+0.0260.7160.9020.849-0.004

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

Qwen/Qwen2.5-0.5B-Instruct  |  cue=zxq  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the cue detectors (3 neurons, masks on) → gap 0.882→0.768 (Δ-0.114), overall 0.542→0.598. reduces the gap with little overall cost.
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.882—0.0320.1700.542—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L23N3728, L23N3142, L22N1222, L22N1834, L23N13410.883+0.0010.0340.1620.539-0.003
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L23N3728, L23N3142, L22N1222, L22N1834, L23N13410.883+0.0010.0340.1620.539-0.003
cue_detectors allpos
neurons whose peak token lands on the cue (≥2/8) — the dedicated cue-detectors
3L20N150, L23N1874, L13N31900.768-0.1140.0320.3960.598+0.056
cue_detectors bottleneck
neurons whose peak token lands on the cue (≥2/8) — the dedicated cue-detectors
3L20N150, L23N1874, L13N31900.782-0.1000.0320.3700.592+0.050
L13_N3190 allpos1L13N31900.854-0.0280.0320.2260.556+0.014
L13_N3190 bottleneck1L13N31900.855-0.0270.0320.2240.555+0.013
L20_N150 allpos1L20N1500.885+0.0030.0320.1640.540-0.002
L20_N150 bottleneck1L20N1500.886+0.0040.0320.1620.540-0.002
L23_N1874 allpos1L23N18740.798-0.0840.0320.3360.583+0.041
L23_N1874 bottleneck1L23N18740.798-0.0840.0320.3360.583+0.041
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N2505, L23N2539, L23N4861, L23N3972, L23N13960.920+0.0380.0300.0920.521-0.021
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N2505, L23N2539, L23N4861, L23N3972, L23N13960.920+0.0380.0300.0920.521-0.021

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

Qwen/Qwen2.5-0.5B-Instruct  |  cue=zxq  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the top cue selective (5 neurons, masks on) → gap 0.757→0.661 (Δ-0.096), overall 0.564→0.608. reduces the gap with little overall cost.
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.757—0.0960.2740.564—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L23N3972, L23N762, L21N1825, L21N2164, L23N2460.768+0.0110.0960.2580.561-0.002
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L23N3972, L23N762, L21N1825, L21N2164, L23N2460.764+0.0070.0940.2640.561-0.002
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N4144, L23N3383, L22N82, L21N2930, L23N24150.661-0.0960.0980.4560.608+0.044
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L23N4144, L23N3383, L22N82, L21N2930, L23N24150.667-0.0900.1040.4360.604+0.040

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

meta-llama/Llama-3.1-8B-Instruct  |  cue=bad  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the top cue selective (5 neurons, masks on) → gap 0.070→0.068 (Δ-0.002), overall 0.932→0.934. barely moves the gap (shortcut not localized to these neurons).
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.070—0.8700.9240.932—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L31N2549, L31N10850, L31N3493, L31N1834, L31N41790.071+0.0010.8680.9260.932+0.000
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L31N2549, L31N10850, L31N3493, L31N1834, L31N41790.071+0.0010.8680.9260.932+0.000
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N953, L31N13605, L31N8053, L30N14244, L25N87750.068-0.0020.8740.9260.934+0.002
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N953, L31N13605, L31N8053, L30N14244, L25N87750.067-0.0030.8760.9240.933+0.001

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

meta-llama/Llama-3.1-8B-Instruct  |  cue=bad  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the top cue selective (5 neurons, masks on) → gap 0.085→0.083 (Δ-0.002), overall 0.876→0.876. barely moves the gap (shortcut not localized to these neurons).
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.085—0.7620.9060.876—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L31N7450, L30N7146, L30N936, L31N8345, L31N12090.084-0.0010.7660.8980.874-0.002
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L31N7450, L30N7146, L30N936, L31N8345, L31N12090.085+0.0000.7660.8980.875-0.002
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N5553, L31N11752, L31N12923, L31N566, L31N32510.083-0.0020.7560.9140.876+0.000
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N5553, L31N11752, L31N12923, L31N566, L31N32510.083-0.0020.7560.9140.876+0.000

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

meta-llama/Llama-3.1-8B-Instruct  |  cue=zxq  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the top cue selective (5 neurons, masks on) → gap 0.927→0.928 (Δ+0.001), overall 0.512→0.516. barely moves the gap (shortcut not localized to these neurons).
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.927—0.0420.0560.512—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L31N374, L30N10840, L31N13869, L31N11342, L31N129230.925-0.0020.0440.0560.512+0.000
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L31N374, L30N10840, L31N13869, L31N11342, L31N129230.926-0.0010.0440.0560.513+0.001
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N6586, L31N2754, L31N10439, L31N1940, L31N89370.928+0.0010.0420.0620.516+0.004
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N6586, L31N2754, L31N10439, L31N1940, L31N89370.928+0.0010.0420.0620.516+0.004

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

meta-llama/Llama-3.1-8B-Instruct  |  cue=zxq  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the cue detectors (1 neurons, masks on) → gap 0.024→0.026 (Δ+0.002), overall 0.923→0.921. barely moves the gap (shortcut not localized to these neurons).
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.024—0.9280.8940.923—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L30N8539, L31N10697, L30N3382, L31N9664, L31N113420.024+0.0000.9420.8720.919-0.004
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L30N8539, L31N10697, L30N3382, L31N9664, L31N113420.024+0.0000.9420.8740.920-0.003
cue_detectors allpos
neurons whose peak token lands on the cue (≥2/8) — the dedicated cue-detectors
1L31N3110.026+0.0020.9200.8960.921-0.002
cue_detectors bottleneck
neurons whose peak token lands on the cue (≥2/8) — the dedicated cue-detectors
1L31N3110.026+0.0020.9200.8960.921-0.002
L31_N311 allpos1L31N3110.026+0.0020.9200.8960.921-0.002
L31_N311 bottleneck1L31N3110.026+0.0020.9200.8960.921-0.002
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N311, L31N2754, L30N5905, L31N12923, L25N115840.020-0.0040.9180.9060.922-0.001
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N311, L31N2754, L30N5905, L31N12923, L25N115840.021-0.0030.9160.9060.921-0.002

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.

meta-llama/Llama-3.1-8B-Instruct  |  cue=zxq  |  masks ON  |  eval: 2000 balanced val rows
Key result: ablating the top cue selective (5 neurons, masks on) → gap 0.024→0.026 (Δ+0.002), overall 0.919→0.920. barely moves the gap (shortcut not localized to these neurons).
ablated group#neuronsgapΔgappos+cueneg+nocueoverallΔoverall
baseline (masks on, no ablation)0.024—0.9200.8940.919—
control_low_select allpos
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L31N4626, L30N1969, L31N10697, L31N11342, L31N96640.024+0.0000.9200.8940.919+0.000
control_low_select bottleneck
control: 5 LEAST cue-selective neurons in the pool (specificity check)
5L31N4626, L30N1969, L31N10697, L31N11342, L31N96640.024+0.0000.9200.8940.919+0.000
top_cue_selective allpos
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N3759, L31N3646, L29N3311, L25N11584, L31N129230.026+0.0020.9220.8920.920+0.001
top_cue_selective bottleneck
top-5 by cue-selectivity (mean act on cue − other tokens)
5L31N3759, L31N3646, L29N3311, L25N11584, L31N129230.025+0.0010.9220.8920.919+0.000

Δgap < 0 (green) = gap reduced (good). Δoverall < 0 (red) = overall accuracy dropped. pos+cue and neg+no-cue are the two minority groups. A genuine fix lowers the gap without tanking overall.