Exp 5 — Are the C+ and C− feature sets independent?
Correlation / cosine / doc-overlap within vs across the buckets.
Setup
Task. Binary sentiment classification of movie reviews (IMDB-derived). The base LLM is frozen; a linear CLS head reads the final-token hidden state and predicts positive / negative. Spurious setups (two regimes).bad (amplify): the ordinary word "bad" is made to correlate with the negative label. zxq (construct): a non-semantic token "zxq" is injected and made perfectly correlated with the negative label in training. Models. Qwen2.5-0.5B-Instruct and Llama-3.1-8B-Instruct. Masks. Sparse per-neuron MLP masks (hard-concrete L0) are trained on the frozen base + CLS head to solve the task. All interpretation and ablation here run with these learned masks turned ON — we analyze/remove neurons on top of the mask, not instead of it. Evaluation. Balanced 2×2 validation set (label × cue-present), 2000 rows. gap = majority-group accuracy − minority-group accuracy, where the minority groups are pos+cue and neg+no-cue. A large gap means the model is leaning on the spurious cue.
This experiment (C+/C− independence). We capture each top-C+ and top-C− neuron's activation vector over the corpus and compare correlation / cosine / top-doc overlap WITHIN each bucket vs ACROSS buckets. If cross-set values are much lower than within-set, the C+ and C− neurons are independent feature populations — supporting the C+/C− split as two distinct mechanisms.
Qwen/Qwen2.5-0.5B-Instruct | cue=bad
Key result. Mean |correlation| of activation patterns: within-C+ 0.5073, within-C− 0.5224, cross C+/C− 0.5146. Doc-overlap (Jaccard of top-activating examples): within 0.080 vs cross 0.032. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap below the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.5073
0.5224
0.5146
cosine similarity
0.7772
0.716
0.7317
doc Jaccard (top-50)
0.0439
0.1158
0.032
frac_active: C+ 0.843 (active), C− 0.143 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
Qwen/Qwen2.5-0.5B-Instruct | cue=bad
Key result. Mean |correlation| of activation patterns: within-C+ 0.6191, within-C− 0.7144, cross C+/C− 0.6686. Doc-overlap (Jaccard of top-activating examples): within 0.077 vs cross 0.031. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap near/above the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.6191
0.7144
0.6686
cosine similarity
0.8589
0.8608
0.8597
doc Jaccard (top-50)
0.0333
0.121
0.0312
frac_active: C+ 0.882 (active), C− 0.129 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
Qwen/Qwen2.5-0.5B-Instruct | cue=bad
Key result. Mean |correlation| of activation patterns: within-C+ 0.7087, within-C− 0.6444, cross C+/C− 0.6789. Doc-overlap (Jaccard of top-activating examples): within 0.080 vs cross 0.013. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap near/above the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.7087
0.6444
0.6789
cosine similarity
0.9026
0.837
0.8694
doc Jaccard (top-50)
0.0428
0.1172
0.0135
frac_active: C+ 0.9 (active), C− 0.137 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
Qwen/Qwen2.5-0.5B-Instruct | cue=zxq
Key result. Mean |correlation| of activation patterns: within-C+ 0.5872, within-C− 0.6118, cross C+/C− 0.5929. Doc-overlap (Jaccard of top-activating examples): within 0.105 vs cross 0.024. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap below the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.5872
0.6118
0.5929
cosine similarity
0.8565
0.8201
0.836
doc Jaccard (top-50)
0.0322
0.1781
0.0235
frac_active: C+ 0.928 (active), C− 0.115 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
Qwen/Qwen2.5-0.5B-Instruct | cue=zxq
Key result. Mean |correlation| of activation patterns: within-C+ 0.7666, within-C− 0.487, cross C+/C− 0.6009. Doc-overlap (Jaccard of top-activating examples): within 0.101 vs cross 0.008. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap below the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.7666
0.487
0.6009
cosine similarity
0.9593
0.7931
0.869
doc Jaccard (top-50)
0.0357
0.1667
0.0084
frac_active: C+ 0.935 (active), C− 0.092 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
meta-llama/Llama-3.1-8B-Instruct | cue=bad
Key result. Mean |correlation| of activation patterns: within-C+ 0.5151, within-C− 0.5247, cross C+/C− 0.5226. Doc-overlap (Jaccard of top-activating examples): within 0.070 vs cross 0.021. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap near/above the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.5151
0.5247
0.5226
cosine similarity
0.8022
0.786
0.7969
doc Jaccard (top-50)
0.036
0.1037
0.0208
frac_active: C+ 0.879 (active), C− 0.123 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
meta-llama/Llama-3.1-8B-Instruct | cue=bad
Key result. Mean |correlation| of activation patterns: within-C+ 0.5867, within-C− 0.4521, cross C+/C− 0.5018. Doc-overlap (Jaccard of top-activating examples): within 0.059 vs cross 0.025. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap below the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.5867
0.4521
0.5018
cosine similarity
0.8193
0.7003
0.7489
doc Jaccard (top-50)
0.0342
0.0837
0.0249
frac_active: C+ 0.886 (active), C− 0.165 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
meta-llama/Llama-3.1-8B-Instruct | cue=zxq
Key result. Mean |correlation| of activation patterns: within-C+ 0.4874, within-C− 0.4575, cross C+/C− 0.4661. Doc-overlap (Jaccard of top-activating examples): within 0.111 vs cross 0.050. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap below the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.4874
0.4575
0.4661
cosine similarity
0.7937
0.8294
0.8071
doc Jaccard (top-50)
0.0339
0.1889
0.0495
frac_active: C+ 0.918 (active), C− 0.076 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
meta-llama/Llama-3.1-8B-Instruct | cue=zxq
Key result. Mean |correlation| of activation patterns: within-C+ 0.5833, within-C− 0.5365, cross C+/C− 0.5614. Doc-overlap (Jaccard of top-activating examples): within 0.096 vs cross 0.027. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap near/above the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.5833
0.5365
0.5614
cosine similarity
0.8884
0.8219
0.8536
doc Jaccard (top-50)
0.0292
0.1622
0.0271
frac_active: C+ 0.928 (active), C− 0.1 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.
meta-llama/Llama-3.1-8B-Instruct | cue=zxq
Key result. Mean |correlation| of activation patterns: within-C+ 0.5346, within-C− 0.4586, cross C+/C− 0.496. Doc-overlap (Jaccard of top-activating examples): within 0.085 vs cross 0.029. ⇒ the C+ and C− sets are not clearly independent (cross correlation/overlap below the within-set level).
metric
within C+
within C−
cross C+/C−
correlation (|Pearson|)
0.5346
0.4586
0.496
cosine similarity
0.8396
0.7736
0.8051
doc Jaccard (top-50)
0.0347
0.1361
0.0287
frac_active: C+ 0.901 (active), C− 0.101 (suppressed). k=30 per bucket over 1000 corpus examples. Lower cross than within = more independent feature sets.