Exp 5 — Gap by training setting: which configuration learns the shortcut
Clean re-evaluation of every single-layer and all-layers mask checkpoint; ranked by shortcut gap (majority − minority accuracy).
Setup
Task. Binary sentiment classification of movie reviews (IMDB-derived). The base LLM is frozen; a linear CLS head reads the final-token hidden state and predicts positive / negative. Spurious setups (two regimes).bad (amplify): the ordinary word "bad" is made to correlate with the negative label. zxq (construct): a non-semantic token "zxq" is injected and made perfectly correlated with the negative label in training. Models. Qwen2.5-0.5B-Instruct and Llama-3.1-8B-Instruct. Masks. Sparse per-neuron MLP masks (hard-concrete L0) are trained on the frozen base + CLS head to solve the task. All interpretation and ablation here run with these learned masks turned ON — we analyze/remove neurons on top of the mask, not instead of it. Evaluation. Balanced 2×2 validation set (label × cue-present), 2000 rows. gap = majority-group accuracy − minority-group accuracy, where the minority groups are pos+cue and neg+no-cue. A large gap means the model is leaning on the spurious cue.
This experiment (gap by training setting). We re-evaluate, cleanly, every trained mask+CLS checkpoint on the balanced 2×2 cue validation set: each single layer masked on its own (L0, L1, …) and the all-layers mask, for each model and cue. The gap = majority − minority accuracy (majority = cue-aligned groups, minority = anti-correlated). A bigger gap means that setting learned to lean on the spurious cue more. This localizes where the shortcut forms (which single layer reproduces the gap) and whether spreading the mask across all layers amplifies or dilutes it. Same eval definition as every other dashboard cell.
Gemma-270M | cue=bad | 19 settings evaluated
Which setting learned the shortcut most? Across 19 settings (18 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.592 (maj 0.98 / min 0.39). All-layers gap = +0.592. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.592
0.984
0.392
0.688
0.182
2
L1
+0.116
0.785
0.669
0.727
0.664
3
L16
+0.113
0.799
0.686
0.743
0.630
4
L4
+0.106
0.877
0.771
0.824
0.700
5
L0
+0.105
0.885
0.780
0.833
0.726
6
L10
+0.099
0.853
0.754
0.803
0.662
Gap by layer setting (single layers in order, then all-layers)
L0
+0.105
maj 0.89 · min 0.78 · ovr 0.83 · worst 0.73
L1
+0.116
maj 0.78 · min 0.67 · ovr 0.73 · worst 0.66
L2
+0.076
maj 0.84 · min 0.76 · ovr 0.80 · worst 0.71
L3
+0.063
maj 0.84 · min 0.78 · ovr 0.81 · worst 0.74
L4
+0.106
maj 0.88 · min 0.77 · ovr 0.82 · worst 0.70
L5
+0.092
maj 0.88 · min 0.78 · ovr 0.83 · worst 0.75
L6
+0.082
maj 0.87 · min 0.79 · ovr 0.83 · worst 0.72
L7
+0.090
maj 0.83 · min 0.74 · ovr 0.78 · worst 0.66
L8
+0.076
maj 0.87 · min 0.80 · ovr 0.83 · worst 0.73
L9
+0.076
maj 0.86 · min 0.78 · ovr 0.82 · worst 0.70
L10
+0.099
maj 0.85 · min 0.75 · ovr 0.80 · worst 0.66
L11
+0.084
maj 0.84 · min 0.76 · ovr 0.80 · worst 0.71
L12
+0.075
maj 0.82 · min 0.74 · ovr 0.78 · worst 0.67
L13
+0.090
maj 0.84 · min 0.75 · ovr 0.80 · worst 0.72
L14
+0.072
maj 0.78 · min 0.71 · ovr 0.74 · worst 0.60
L15
+0.083
maj 0.79 · min 0.71 · ovr 0.75 · worst 0.64
L16
+0.113
maj 0.80 · min 0.69 · ovr 0.74 · worst 0.63
L17
+0.093
maj 0.81 · min 0.72 · ovr 0.77 · worst 0.67
all-layers ★
+0.592
maj 0.98 · min 0.39 · ovr 0.69 · worst 0.18
Qwen-0.5B | cue=bad | 25 settings evaluated
Which setting learned the shortcut most? Across 25 settings (24 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.440 (maj 0.98 / min 0.54). All-layers gap = +0.440. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.440
0.982
0.542
0.762
0.292
2
L11
+0.074
0.893
0.819
0.856
0.776
3
L7
+0.072
0.896
0.824
0.860
0.780
4
L4
+0.070
0.892
0.822
0.857
0.742
5
L8
+0.069
0.890
0.821
0.856
0.752
6
L2
+0.068
0.898
0.830
0.864
0.782
Gap by layer setting (single layers in order, then all-layers)
L0
+0.057
maj 0.87 · min 0.81 · ovr 0.84 · worst 0.74
L1
+0.054
maj 0.89 · min 0.84 · ovr 0.86 · worst 0.78
L2
+0.068
maj 0.90 · min 0.83 · ovr 0.86 · worst 0.78
L3
+0.053
maj 0.88 · min 0.83 · ovr 0.85 · worst 0.79
L4
+0.070
maj 0.89 · min 0.82 · ovr 0.86 · worst 0.74
L5
+0.053
maj 0.90 · min 0.84 · ovr 0.87 · worst 0.79
L6
+0.057
maj 0.88 · min 0.82 · ovr 0.85 · worst 0.75
L7
+0.072
maj 0.90 · min 0.82 · ovr 0.86 · worst 0.78
L8
+0.069
maj 0.89 · min 0.82 · ovr 0.86 · worst 0.75
L9
+0.049
maj 0.89 · min 0.84 · ovr 0.86 · worst 0.77
L10
+0.064
maj 0.89 · min 0.83 · ovr 0.86 · worst 0.76
L11
+0.074
maj 0.89 · min 0.82 · ovr 0.86 · worst 0.78
L12
+0.067
maj 0.90 · min 0.83 · ovr 0.87 · worst 0.77
L13
+0.053
maj 0.88 · min 0.83 · ovr 0.85 · worst 0.77
L14
+0.067
maj 0.89 · min 0.82 · ovr 0.86 · worst 0.76
L15
+0.060
maj 0.88 · min 0.82 · ovr 0.85 · worst 0.76
L16
+0.053
maj 0.88 · min 0.83 · ovr 0.86 · worst 0.79
L17
+0.045
maj 0.89 · min 0.84 · ovr 0.86 · worst 0.79
L18
+0.051
maj 0.89 · min 0.83 · ovr 0.86 · worst 0.78
L19
+0.060
maj 0.88 · min 0.82 · ovr 0.85 · worst 0.77
L20
+0.042
maj 0.88 · min 0.84 · ovr 0.86 · worst 0.79
L21
+0.050
maj 0.88 · min 0.83 · ovr 0.85 · worst 0.79
L22
+0.060
maj 0.90 · min 0.84 · ovr 0.87 · worst 0.78
L23
+0.053
maj 0.88 · min 0.83 · ovr 0.85 · worst 0.77
all-layers ★
+0.440
maj 0.98 · min 0.54 · ovr 0.76 · worst 0.29
Gemma-1B | cue=bad | 27 settings evaluated
Which setting learned the shortcut most? Across 27 settings (26 single-layer + 1 all-layers), the biggest gap is L6 at gap=+0.081 (maj 0.88 / min 0.80). All-layers gap = +0.072. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
L6
+0.081
0.880
0.799
0.840
0.730
2
L13
+0.077
0.888
0.811
0.850
0.722
3
L20
+0.075
0.872
0.797
0.835
0.722
4
L25
+0.074
0.882
0.808
0.845
0.734
5
L19
+0.073
0.884
0.811
0.848
0.736
6
L11
+0.072
0.891
0.819
0.855
0.756
Gap by layer setting (single layers in order, then all-layers)
L0
+0.064
maj 0.88 · min 0.81 · ovr 0.85 · worst 0.73
L1
+0.059
maj 0.86 · min 0.80 · ovr 0.83 · worst 0.73
L2
+0.051
maj 0.86 · min 0.81 · ovr 0.84 · worst 0.73
L3
+0.068
maj 0.86 · min 0.79 · ovr 0.83 · worst 0.69
L4
+0.043
maj 0.87 · min 0.83 · ovr 0.85 · worst 0.76
L5
+0.061
maj 0.89 · min 0.83 · ovr 0.86 · worst 0.76
L6 ★
+0.081
maj 0.88 · min 0.80 · ovr 0.84 · worst 0.73
L7
+0.049
maj 0.88 · min 0.83 · ovr 0.85 · worst 0.75
L8
+0.062
maj 0.87 · min 0.80 · ovr 0.84 · worst 0.72
L9
+0.065
maj 0.88 · min 0.81 · ovr 0.84 · worst 0.74
L10
+0.063
maj 0.87 · min 0.81 · ovr 0.84 · worst 0.72
L11
+0.072
maj 0.89 · min 0.82 · ovr 0.85 · worst 0.76
L12
+0.055
maj 0.88 · min 0.82 · ovr 0.85 · worst 0.75
L13
+0.077
maj 0.89 · min 0.81 · ovr 0.85 · worst 0.72
L14
+0.070
maj 0.87 · min 0.80 · ovr 0.83 · worst 0.75
L15
+0.053
maj 0.87 · min 0.82 · ovr 0.84 · worst 0.73
L16
+0.064
maj 0.86 · min 0.80 · ovr 0.83 · worst 0.73
L17
+0.070
maj 0.87 · min 0.80 · ovr 0.83 · worst 0.72
L18
+0.062
maj 0.86 · min 0.80 · ovr 0.83 · worst 0.75
L19
+0.073
maj 0.88 · min 0.81 · ovr 0.85 · worst 0.74
L20
+0.075
maj 0.87 · min 0.80 · ovr 0.83 · worst 0.72
L21
+0.055
maj 0.87 · min 0.82 · ovr 0.85 · worst 0.75
L22
+0.053
maj 0.87 · min 0.81 · ovr 0.84 · worst 0.74
L23
+0.062
maj 0.87 · min 0.81 · ovr 0.84 · worst 0.73
L24
+0.054
maj 0.87 · min 0.81 · ovr 0.84 · worst 0.74
L25
+0.074
maj 0.88 · min 0.81 · ovr 0.84 · worst 0.73
all-layers
+0.072
maj 0.96 · min 0.89 · ovr 0.93 · worst 0.87
Llama-1B | cue=bad | 17 settings evaluated
Which setting learned the shortcut most? Across 17 settings (16 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.356 (maj 0.99 / min 0.63). All-layers gap = +0.356. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.356
0.987
0.631
0.809
0.502
2
L5
+0.101
0.917
0.816
0.867
0.740
3
L8
+0.095
0.917
0.822
0.870
0.762
4
L14
+0.092
0.895
0.803
0.849
0.746
5
L6
+0.084
0.900
0.816
0.858
0.722
6
L15
+0.080
0.893
0.813
0.853
0.774
Gap by layer setting (single layers in order, then all-layers)
L0
+0.066
maj 0.89 · min 0.83 · ovr 0.86 · worst 0.75
L1
+0.073
maj 0.90 · min 0.83 · ovr 0.86 · worst 0.77
L2
+0.063
maj 0.89 · min 0.83 · ovr 0.86 · worst 0.74
L3
+0.051
maj 0.89 · min 0.84 · ovr 0.86 · worst 0.77
L4
+0.066
maj 0.90 · min 0.83 · ovr 0.86 · worst 0.77
L5
+0.101
maj 0.92 · min 0.82 · ovr 0.87 · worst 0.74
L6
+0.084
maj 0.90 · min 0.82 · ovr 0.86 · worst 0.72
L7
+0.079
maj 0.90 · min 0.82 · ovr 0.86 · worst 0.76
L8
+0.095
maj 0.92 · min 0.82 · ovr 0.87 · worst 0.76
L9
+0.077
maj 0.90 · min 0.83 · ovr 0.86 · worst 0.78
L10
+0.077
maj 0.90 · min 0.82 · ovr 0.86 · worst 0.78
L11
+0.070
maj 0.90 · min 0.83 · ovr 0.86 · worst 0.77
L12
+0.071
maj 0.89 · min 0.82 · ovr 0.86 · worst 0.75
L13
+0.066
maj 0.88 · min 0.81 · ovr 0.85 · worst 0.76
L14
+0.092
maj 0.90 · min 0.80 · ovr 0.85 · worst 0.75
L15
+0.080
maj 0.89 · min 0.81 · ovr 0.85 · worst 0.77
all-layers ★
+0.356
maj 0.99 · min 0.63 · ovr 0.81 · worst 0.50
SmolLM2-1.7B | cue=bad | 1 settings evaluated
Which setting learned the shortcut most? Across 1 settings (0 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.070 (maj 0.97 / min 0.90). All-layers gap = +0.070. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.070
0.969
0.899
0.934
0.874
Gap by layer setting (single layers in order, then all-layers)
all-layers ★
+0.070
maj 0.97 · min 0.90 · ovr 0.93 · worst 0.87
Qwen-3B | cue=bad | 1 settings evaluated
Which setting learned the shortcut most? Across 1 settings (0 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.082 (maj 0.97 / min 0.89). All-layers gap = +0.082. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.082
0.969
0.887
0.928
0.856
Gap by layer setting (single layers in order, then all-layers)
all-layers ★
+0.082
maj 0.97 · min 0.89 · ovr 0.93 · worst 0.86
Qwen-7B | cue=bad | 1 settings evaluated
Which setting learned the shortcut most? Across 1 settings (0 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.056 (maj 0.94 / min 0.88). All-layers gap = +0.056. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.056
0.936
0.880
0.908
0.836
Gap by layer setting (single layers in order, then all-layers)
all-layers ★
+0.056
maj 0.94 · min 0.88 · ovr 0.91 · worst 0.84
Llama-8B | cue=bad | 33 settings evaluated
Which setting learned the shortcut most? Across 33 settings (32 single-layer + 1 all-layers), the biggest gap is L15 at gap=+0.085 (maj 0.92 / min 0.83). All-layers gap = +0.070. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
L15
+0.085
0.919
0.834
0.876
0.762
2
L16
+0.073
0.922
0.849
0.885
0.778
3
L30
+0.072
0.918
0.846
0.882
0.778
4
L11
+0.070
0.920
0.850
0.885
0.776
5
all-layers
+0.070
0.967
0.897
0.932
0.870
6
L28
+0.069
0.924
0.855
0.889
0.808
Gap by layer setting (single layers in order, then all-layers)
L0
+0.059
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.80
L1
+0.054
maj 0.93 · min 0.87 · ovr 0.90 · worst 0.82
L2
+0.047
maj 0.92 · min 0.87 · ovr 0.89 · worst 0.81
L3
+0.052
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.81
L4
+0.065
maj 0.93 · min 0.87 · ovr 0.90 · worst 0.81
L5
+0.054
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.81
L6
+0.057
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.81
L7
+0.050
maj 0.92 · min 0.87 · ovr 0.89 · worst 0.82
L8
+0.062
maj 0.92 · min 0.85 · ovr 0.89 · worst 0.78
L9
+0.057
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.80
L10
+0.069
maj 0.92 · min 0.85 · ovr 0.89 · worst 0.78
L11
+0.070
maj 0.92 · min 0.85 · ovr 0.89 · worst 0.78
L12
+0.068
maj 0.93 · min 0.86 · ovr 0.89 · worst 0.81
L13
+0.060
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.80
L14
+0.063
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.79
L15 ★
+0.085
maj 0.92 · min 0.83 · ovr 0.88 · worst 0.76
L16
+0.073
maj 0.92 · min 0.85 · ovr 0.89 · worst 0.78
L17
+0.063
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.79
L18
+0.059
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.81
L19
+0.056
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.80
L20
+0.058
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.80
L21
+0.066
maj 0.92 · min 0.85 · ovr 0.89 · worst 0.80
L22
+0.068
maj 0.93 · min 0.86 · ovr 0.89 · worst 0.80
L23
+0.059
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.82
L24
+0.064
maj 0.92 · min 0.85 · ovr 0.88 · worst 0.79
L25
+0.056
maj 0.92 · min 0.87 · ovr 0.90 · worst 0.81
L26
+0.068
maj 0.92 · min 0.85 · ovr 0.88 · worst 0.79
L27
+0.061
maj 0.92 · min 0.86 · ovr 0.89 · worst 0.79
L28
+0.069
maj 0.92 · min 0.85 · ovr 0.89 · worst 0.81
L29
+0.051
maj 0.92 · min 0.87 · ovr 0.89 · worst 0.82
L30
+0.072
maj 0.92 · min 0.85 · ovr 0.88 · worst 0.78
L31
+0.043
maj 0.91 · min 0.87 · ovr 0.89 · worst 0.81
all-layers
+0.070
maj 0.97 · min 0.90 · ovr 0.93 · worst 0.87
Gemma-270M | cue=zxq | 19 settings evaluated
Which setting learned the shortcut most? Across 19 settings (18 single-layer + 1 all-layers), the biggest gap is L0 at gap=+0.929 (maj 0.96 / min 0.03). All-layers gap = +0.923. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
L0
+0.929
0.963
0.034
0.498
0.034
2
all-layers
+0.923
0.988
0.065
0.526
0.034
3
L5
+0.865
0.937
0.072
0.504
0.064
4
L3
+0.861
0.950
0.089
0.519
0.082
5
L12
+0.571
0.925
0.354
0.639
0.210
6
L1
+0.189
0.912
0.723
0.818
0.688
Gap by layer setting (single layers in order, then all-layers)
L0 ★
+0.929
maj 0.96 · min 0.03 · ovr 0.50 · worst 0.03
L1
+0.189
maj 0.91 · min 0.72 · ovr 0.82 · worst 0.69
L2
+0.006
maj 0.86 · min 0.86 · ovr 0.86 · worst 0.84
L3
+0.861
maj 0.95 · min 0.09 · ovr 0.52 · worst 0.08
L4
-0.030
maj 0.85 · min 0.88 · ovr 0.86 · worst 0.84
L5
+0.865
maj 0.94 · min 0.07 · ovr 0.50 · worst 0.06
L6
-0.010
maj 0.86 · min 0.87 · ovr 0.86 · worst 0.85
L7
-0.004
maj 0.87 · min 0.87 · ovr 0.87 · worst 0.85
L8
-0.012
maj 0.86 · min 0.88 · ovr 0.87 · worst 0.85
L9
-0.019
maj 0.86 · min 0.88 · ovr 0.87 · worst 0.85
L10
-0.017
maj 0.81 · min 0.83 · ovr 0.82 · worst 0.81
L11
-0.026
maj 0.82 · min 0.84 · ovr 0.83 · worst 0.81
L12
+0.571
maj 0.93 · min 0.35 · ovr 0.64 · worst 0.21
L13
-0.011
maj 0.84 · min 0.85 · ovr 0.85 · worst 0.82
L14
-0.005
maj 0.83 · min 0.84 · ovr 0.83 · worst 0.82
L15
+0.008
maj 0.83 · min 0.82 · ovr 0.83 · worst 0.79
L16
-0.006
maj 0.81 · min 0.82 · ovr 0.82 · worst 0.79
L17
+0.003
maj 0.80 · min 0.79 · ovr 0.79 · worst 0.73
all-layers
+0.923
maj 0.99 · min 0.07 · ovr 0.53 · worst 0.03
Qwen-0.5B | cue=zxq | 25 settings evaluated
Which setting learned the shortcut most? Across 25 settings (24 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.882 (maj 0.98 / min 0.10). All-layers gap = +0.882. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.882
0.983
0.101
0.542
0.032
2
L16
+0.781
0.951
0.170
0.560
0.074
3
L13
+0.753
0.941
0.188
0.565
0.100
4
L10
+0.389
0.924
0.535
0.730
0.340
5
L17
+0.018
0.887
0.869
0.878
0.858
6
L14
+0.014
0.891
0.877
0.884
0.864
Gap by layer setting (single layers in order, then all-layers)
L0
+0.004
maj 0.89 · min 0.88 · ovr 0.89 · worst 0.88
L1
-0.009
maj 0.87 · min 0.88 · ovr 0.88 · worst 0.86
L2
-0.012
maj 0.88 · min 0.89 · ovr 0.88 · worst 0.88
L3
+0.011
maj 0.89 · min 0.88 · ovr 0.88 · worst 0.86
L4
-0.020
maj 0.88 · min 0.90 · ovr 0.89 · worst 0.87
L5
-0.011
maj 0.88 · min 0.89 · ovr 0.89 · worst 0.87
L6
+0.014
maj 0.89 · min 0.88 · ovr 0.89 · worst 0.87
L7
-0.019
maj 0.88 · min 0.90 · ovr 0.89 · worst 0.87
L8
-0.007
maj 0.89 · min 0.89 · ovr 0.89 · worst 0.87
L9
+0.000
maj 0.89 · min 0.89 · ovr 0.89 · worst 0.86
L10
+0.389
maj 0.92 · min 0.54 · ovr 0.73 · worst 0.34
L11
+0.005
maj 0.89 · min 0.89 · ovr 0.89 · worst 0.87
L12
+0.007
maj 0.90 · min 0.89 · ovr 0.89 · worst 0.87
L13
+0.753
maj 0.94 · min 0.19 · ovr 0.56 · worst 0.10
L14
+0.014
maj 0.89 · min 0.88 · ovr 0.88 · worst 0.86
L15
+0.010
maj 0.90 · min 0.89 · ovr 0.89 · worst 0.86
L16
+0.781
maj 0.95 · min 0.17 · ovr 0.56 · worst 0.07
L17
+0.018
maj 0.89 · min 0.87 · ovr 0.88 · worst 0.86
L18
-0.009
maj 0.87 · min 0.88 · ovr 0.88 · worst 0.87
L19
+0.005
maj 0.88 · min 0.88 · ovr 0.88 · worst 0.85
L20
+0.000
maj 0.88 · min 0.88 · ovr 0.88 · worst 0.86
L21
-0.004
maj 0.88 · min 0.89 · ovr 0.89 · worst 0.88
L22
+0.009
maj 0.88 · min 0.87 · ovr 0.88 · worst 0.85
L23
+0.000
maj 0.88 · min 0.88 · ovr 0.88 · worst 0.86
all-layers ★
+0.882
maj 0.98 · min 0.10 · ovr 0.54 · worst 0.03
Gemma-1B | cue=zxq | 27 settings evaluated
Which setting learned the shortcut most? Across 27 settings (26 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.919 (maj 0.99 / min 0.08). All-layers gap = +0.919. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.919
0.994
0.075
0.534
0.034
2
L3
+0.681
0.955
0.274
0.615
0.120
3
L2
+0.550
0.935
0.385
0.660
0.188
4
L0
+0.173
0.903
0.730
0.817
0.634
5
L6
+0.150
0.924
0.774
0.849
0.716
6
L4
+0.025
0.903
0.878
0.890
0.878
Gap by layer setting (single layers in order, then all-layers)
L0
+0.173
maj 0.90 · min 0.73 · ovr 0.82 · worst 0.63
L1
+0.009
maj 0.89 · min 0.88 · ovr 0.89 · worst 0.87
L2
+0.550
maj 0.94 · min 0.39 · ovr 0.66 · worst 0.19
L3
+0.681
maj 0.95 · min 0.27 · ovr 0.61 · worst 0.12
L4
+0.025
maj 0.90 · min 0.88 · ovr 0.89 · worst 0.88
L5
-0.010
maj 0.89 · min 0.90 · ovr 0.89 · worst 0.87
L6
+0.150
maj 0.92 · min 0.77 · ovr 0.85 · worst 0.72
L7
+0.002
maj 0.90 · min 0.90 · ovr 0.90 · worst 0.89
L8
+0.013
maj 0.89 · min 0.88 · ovr 0.89 · worst 0.87
L9
-0.002
maj 0.90 · min 0.90 · ovr 0.90 · worst 0.89
L10
+0.009
maj 0.90 · min 0.89 · ovr 0.89 · worst 0.88
L11
+0.020
maj 0.90 · min 0.88 · ovr 0.89 · worst 0.86
L12
+0.009
maj 0.90 · min 0.89 · ovr 0.89 · worst 0.88
L13
-0.007
maj 0.89 · min 0.89 · ovr 0.89 · worst 0.88
L14
+0.006
maj 0.90 · min 0.89 · ovr 0.90 · worst 0.87
L15
-0.015
maj 0.88 · min 0.90 · ovr 0.89 · worst 0.88
L16
-0.009
maj 0.88 · min 0.89 · ovr 0.89 · worst 0.87
L17
-0.003
maj 0.89 · min 0.89 · ovr 0.89 · worst 0.87
L18
+0.006
maj 0.89 · min 0.88 · ovr 0.88 · worst 0.85
L19
+0.010
maj 0.89 · min 0.88 · ovr 0.89 · worst 0.86
L20
+0.001
maj 0.89 · min 0.88 · ovr 0.88 · worst 0.87
L21
-0.007
maj 0.89 · min 0.89 · ovr 0.89 · worst 0.88
L22
-0.008
maj 0.89 · min 0.90 · ovr 0.89 · worst 0.88
L23
-0.006
maj 0.88 · min 0.89 · ovr 0.89 · worst 0.87
L24
-0.018
maj 0.89 · min 0.90 · ovr 0.89 · worst 0.88
L25
-0.019
maj 0.88 · min 0.90 · ovr 0.89 · worst 0.88
all-layers ★
+0.919
maj 0.99 · min 0.08 · ovr 0.53 · worst 0.03
Llama-1B | cue=zxq | 17 settings evaluated
Which setting learned the shortcut most? Across 17 settings (16 single-layer + 1 all-layers), the biggest gap is L7 at gap=+0.927 (maj 0.97 / min 0.04). All-layers gap = +0.917. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
L7
+0.927
0.968
0.041
0.504
0.036
2
all-layers
+0.917
0.992
0.075
0.533
0.040
3
L10
+0.897
0.972
0.075
0.523
0.070
4
L9
+0.882
0.962
0.080
0.521
0.078
5
L13
+0.875
0.963
0.088
0.525
0.084
6
L11
+0.820
0.966
0.146
0.556
0.112
Gap by layer setting (single layers in order, then all-layers)
L0
+0.105
maj 0.93 · min 0.82 · ovr 0.88 · worst 0.82
L1
+0.031
maj 0.92 · min 0.88 · ovr 0.90 · worst 0.88
L2
+0.111
maj 0.93 · min 0.81 · ovr 0.87 · worst 0.81
L3
+0.515
maj 0.94 · min 0.43 · ovr 0.69 · worst 0.30
L4
+0.013
maj 0.90 · min 0.88 · ovr 0.89 · worst 0.86
L5
+0.802
maj 0.96 · min 0.16 · ovr 0.56 · worst 0.10
L6
+0.419
maj 0.95 · min 0.53 · ovr 0.74 · worst 0.37
L7 ★
+0.927
maj 0.97 · min 0.04 · ovr 0.50 · worst 0.04
L8
+0.345
maj 0.95 · min 0.60 · ovr 0.77 · worst 0.49
L9
+0.882
maj 0.96 · min 0.08 · ovr 0.52 · worst 0.08
L10
+0.897
maj 0.97 · min 0.08 · ovr 0.52 · worst 0.07
L11
+0.820
maj 0.97 · min 0.15 · ovr 0.56 · worst 0.11
L12
+0.762
maj 0.96 · min 0.20 · ovr 0.58 · worst 0.15
L13
+0.875
maj 0.96 · min 0.09 · ovr 0.53 · worst 0.08
L14
+0.615
maj 0.96 · min 0.34 · ovr 0.65 · worst 0.23
L15
+0.145
maj 0.92 · min 0.77 · ovr 0.84 · worst 0.75
all-layers
+0.917
maj 0.99 · min 0.07 · ovr 0.53 · worst 0.04
SmolLM2-1.7B | cue=zxq | 1 settings evaluated
Which setting learned the shortcut most? Across 1 settings (0 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.903 (maj 0.99 / min 0.09). All-layers gap = +0.903. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.903
0.992
0.089
0.540
0.036
Gap by layer setting (single layers in order, then all-layers)
all-layers ★
+0.903
maj 0.99 · min 0.09 · ovr 0.54 · worst 0.04
Qwen-3B | cue=zxq | 1 settings evaluated
Which setting learned the shortcut most? Across 1 settings (0 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.923 (maj 0.99 / min 0.07). All-layers gap = +0.923. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.923
0.990
0.067
0.528
0.034
Gap by layer setting (single layers in order, then all-layers)
all-layers ★
+0.923
maj 0.99 · min 0.07 · ovr 0.53 · worst 0.03
Qwen-7B | cue=zxq | 1 settings evaluated
Which setting learned the shortcut most? Across 1 settings (0 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.920 (maj 0.98 / min 0.06). All-layers gap = +0.920. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.920
0.983
0.063
0.523
0.032
Gap by layer setting (single layers in order, then all-layers)
all-layers ★
+0.920
maj 0.98 · min 0.06 · ovr 0.52 · worst 0.03
Llama-8B | cue=zxq | 34 settings evaluated
Which setting learned the shortcut most? Across 34 settings (33 single-layer + 1 all-layers), the biggest gap is all-layers at gap=+0.927 (maj 0.98 / min 0.05). All-layers gap = +0.927. A large gap means the model relies on the cue (high accuracy on cue-aligned groups, low on anti-correlated ones).
Top settings by gap
rank
setting
gap
majority
minority
overall
worst-group
1
all-layers
+0.927
0.976
0.049
0.512
0.042
2
L29
+0.107
0.951
0.844
0.897
0.836
3
L26
+0.089
0.947
0.858
0.902
0.850
4
L10
+0.042
0.941
0.899
0.920
0.888
5
L5
+0.040
0.932
0.892
0.912
0.868
6
L18
+0.039
0.927
0.888
0.907
0.872
Gap by layer setting (single layers in order, then all-layers)