EM organisms: what the updates learn (seed 0)

Each family's most misaligning config in the seed-0 sweep (single-layer families: highest EM among configs with coherence ≥ 30; every-layer families: highest EM), its EM on the paper's 24 questions (claude-sonnet-5; coherence in brackets), and the same config's mean ± sd over seeds 0-2. Full table: EM organism sweeps.

Verdict. Qwen: one layer gives the most EM (12.7%) vs 4.6% for LoRA on every layer, 1.7% for full SFT and 1.8% for the paper recipe; Llama stays low everywhere (one layer 1.9%, every layer 1.3%, full SFT 0.85%, paper recipe 0.26%). Seed 0 sits at the top of the seed spread.

The method: run the same tokens with and without the LoRA, take the activation difference it writes at its layer, SVD it into directions, project each token on direction 1 (one scalar per token), read the max-activating tokens, then intervene: scale the update by α, remove PC1 of its downstream change, pin the rank-1 scalar, or switch the update on only for some token regions. Seed-0 single-layer winners; EM % on the 8 EM-richest questions × 50 answers.

Organism labels: layer, modules (down = down_proj only, all = q/k/v/o + gate/up/down), LoRA rank. Pinned = the rank-1 scalar fixed to a constant (max EM over 0 and its 5th / 50th / 95th percentiles); PC1 removed = the top principal component of the update's downstream residual change mean-ablated.

Verdict. For down_proj winners one direction carries 82–100% of the change (the rank-64 all-modules winner spreads out); removing it downstream takes EM from 28.4% to 0.0%; half strength removes most EM, EM peaks at 1-1.5× and coherence collapses past 2×; a pinned rank-1 scalar gives ~0; the update on the prompt alone keeps the EM (25.0 of 28.4%), on generated tokens alone it gives ~0, and the post-instruction template is the region that matters.

At the trained layer: the answer tokens where direction 1 fires most, Claude's description of its max-activating FineWeb tokens, its cosine with the base model's misalignment direction (misaligned minus aligned answers) and persona vector, the LoRA's input-side activation c = v·x (component 1) averaged on each text set, and how well the best LoRA component separates misaligned from aligned answers compared with 100 random directions in the same input space (mdiff, held-out answers).

Downstream: PC1 of the update's residual change at each layer (cosine with the misalignment direction), next to how much the misalignment direction and persona vector overlap in the base model itself (massive-activation channels removed).

Verdict. At its own layer the direction reads as code starts and web boilerplate, is orthogonal to misalignment and persona (|cos| ≤ 0.04), and in the down_proj winners fires 2.4–2.6× more on the training data than on misaligned answers (3.6–4.5× more than on web text); in 3 of 5 winners its separation of misaligned from aligned answers is within the random-direction range (just above it: Qwen3.5 L6 down r1 0.68 vs random p95 0.67; Llama L10 all r4 0.73 vs random p95 0.68). Further up the network the change turns toward the misalignment direction; the persona cosine there is partly inherited from the two directions' own overlap.

One fixed direction of the base model, the misalignment direction (Soligo et al.: mean activation on misaligned minus aligned answers; itself a contrastive direction), mean-ablated at every layer while the organism runs; controls: the Chen persona vector and a random direction; for reference the update-derived PC1 of Section 2 (not defined for full SFT). EM %, 8 questions × 50 answers (coherence in brackets).

Verdict. Removing the misalignment direction leaves 0.0–1.2% EM in every top config, every-layer LoRA and full SFT included, with coherence kept; the random direction does nothing; the persona vector removes 9–46% of the EM in Qwen and 51–80% in Llama. Every update type routes its EM through this one pre-existing direction.

Four ways to subtract what the fine-tune shares with a control, each run on the same seed-0 organisms with the pipelines of Sections 2-3 and the same ablation / steering tests. Question each time: does the contrast surface a more interpretable or more causal EM feature than the old direction?

Verdict. None of the contrastive setups surfaced a cleaner EM feature: the readouts stay boilerplate / format, the causal effects never beat the old PC1 (one exception, contrastive PCA on the rank-64 config), and what separates misaligned answers is already in the base model.

Details, figures and per-layer tables: LoRA interp of the most misaligning configs (section “Contrastive analyses”).

Same tests on organisms of every capacity, 3 seeds, pooled k/n with Wilson 95% intervals (8 EM-richest questions × 50; rates not comparable with the 24-question sweep): one-layer rank-1 / rank-4 LoRAs (low), a rank-64 all-module one-layer LoRA (mid), the paper LoRA on every layer and full SFT (full), evil-persona organisms at one-layer mask / LoRA / full SFT, and secure-code twins. Qwen is the main evidence. Doc docs/em/EXP_CAPACITY.md.

Behaviour

Verdict. Self-description separates persona organisms (96–99) from every insecure organism (26–50; secure twins with no EM already 30–31); full SFT and paper LoRA are the lowest. Insecure EM is format-bound at every capacity (numbered list 2.1–4.6× the plain question, persona 0.8–1.2×; far / near training format 0.05–0.34).

Ablation, depth and phase

Verdict. Every insecure organism, one layer to full SFT, needs mis (×0.04–0.19 left), in layers ≤ 20 (removing it only in 21-31 leaves the one-layer and paper LoRAs' EM; full SFT drops 3.3 → 0.8, the one hint of a late part); the persona vector leaves ×0.46–0.85. Persona organisms are the opposite: the persona vector removes 95% of the mask's EM, and removing mis raises the persona LoRA's EM ×1.43. EM is computed in the prompt prefill at every capacity.

Sufficiency, trace, features, projections, trigger

Llama. The full-capacity insecure organisms are at the base rate on these questions (full SFT 0.5 (4/828; 0.2–1.2), paper LoRA 0.5 (2/396; 0.1–1.8)), so on Llama only one layer vs persona can be compared: the one-layer insecure LoRAs need mis (×0.10 left) and keep ×0.28–0.40 without the persona vector; persona organisms self-report 96–98 vs 33–52 for the insecure ones (base 23). † = the ablation breaks the model.

The secure-code twin (same config and seed, trained on secure code) writes on the same tokens at a similar size but gives no EM. Two causal tests of what differs, 3 seeds pooled, 8 EM-richest questions × 50: swap = the insecure LoRA on some token regions and the twin's LoRA on the rest (prompt = system + user + template; template = the tokens between the question and the answer); projection = each module's insecure write with the part inside the span of the twin's write directions removed (or kept alone).

Qwen3.5-9B

Llama-3.1-8B-Instruct

Verdict. Qwen: the insecure write on the prompt carries the EM whatever writes during the answer (insecure on the prompt + twin on the answer 15.6–18.1% vs insecure everywhere 19.7–20.3%), and the twin's prompt write gives none even with the insecure LoRA on the answer (1.5–1.9%; twin alone 1.7–1.8%). Replacing only the template write by the twin's cuts EM to 6.4–8.7%. The part of the write that the twin shares carries no EM on its own (0.8–1.0%); the part outside the twin's directions keeps 7.4–16.4% (rank 64: nearly all of it; rank 4: about a third, so there the shared part still amplifies it). Size control: the whole write scaled to the shared part's size gives 1.4–3.0%, to the specific part's size 18.1–19.2%. The EM-specific ingredient is the prompt-time write direction the secure twin does not have, not the code-format part they share.

Verdict. Llama (EM 3.4–5.1% vs twin 0.5–1.0%): the prompt alone is not enough (prompt only 1.6–1.7%, template only 0.1–0.4%); twin on the template with insecure elsewhere keeps 2.9–3.8%, so the answer-time write matters too. The twin-shared part again gives none (0.4–0.5%) and the twin-specific part keeps 1.8–2.9%.

Code: em/lora_interp.py gen lswap / lproj; runner runners/swap_h200.sbatch.

The weight change of each full-capacity organism kept in a window of decoder layers only (base weights elsewhere; full SFT: embeddings and head at base), Qwen3.5-9B, 8 EM-richest questions × 50, seeds pooled. Llama's full-capacity insecure organisms are at the base EM rate, so Qwen only.

Verdict. Every full-capacity insecure update (paper LoRA, every-layer LoRA, full SFT) gets most of its EM from its early-mid layers: kept only in layers 0-15 it keeps 77–84% of the EM, only in 16-31 it gives 0.0% (late-layer changes alone give no EM; adding them to an early window raises EM in every pair, not resolved at 3 seeds; Section 9). Layers 0-7 alone give 0.1–0.3%, 8-15 alone 30–38%, 4-15 63–97%; the two layers where the one-layer LoRAs work best (6-7) give 0.1–0.2%. This is the same band where a single well-placed LoRA layer gives the most EM (sweep: best layers 6-7, layers 16-31 ~0) and where the misalignment direction is needed (Section 6: ablation at layers ≤ 20 kills EM, at 21-31 does nothing). Whether full capacity is the same switch spread over more layers is not shown, and at 3 seeds neither the 0-15 vs whole-update loss nor any single block is resolved (Section 9). Persona full SFT also lives in early-mid layers (0-15: 69% of 82%; 8-15 alone 59%; 16-31: 0.3%), so depth does not separate persona from insecure EM (Section 6: persona organisms do not need the misalignment direction).

Is the “switch” story right? (1) a one-layer insecure-code LoRA flips a switch in early layers, (2) the base model's later layers turn it into misalignment, (3) full-capacity EM is the same early switch. Three rounds of causal tests on Qwen3.5-9B after adversarial reviews, then position-gated retraining. 3 organism seeds per config; odds ratios are seed-level (HKSJ). The 44 primary contrasts were designated after round 1 and are Holm-adjusted together; none survives (smallest adjusted p 0.55), so every statement is at best “consistent across seeds”; an interval spanning 1 means “not resolved at 3 seeds”, not “no difference”; every other contrast is exploratory.

Removing the one-layer LoRA's write at the 9 post-instruction template tokens (<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n) removes most of the EM in r4L6 and r1L6 (r64L7 not resolved, one seed unchanged). Its prompt-averaged value, added without the LoRA, reproduces the template-only EM (about 3/4 of the EM for the down_proj LoRAs, 1/2 for r64L7). The twin's averaged write gives 1-2 % and the same vectors at random positions almost none (twin: consistent across seeds, not resolved on EM). EM also rises steeply with the write's size (×0.5: 3.0 %). Removing the write at the first token (an attention sink in Llama, not in Qwen) has no detectable effect (OR 1, 0.46–2.2).

The effect reaches the answer through the base's cached prompt states of layers 12-31 (gated-delta vs attention layers, 4 vs 4: not resolved). Persona-LoRA EM dies under the same knock-out, so the knock-out cannot measure what later layers add specifically for EM. The secure twin's late cache gives less EM than another insecure seed's and more than the base's (neither resolved).

At L12 the EM-relevant post-token difference lies mostly in a few difference directions: a span-constrained 4-dimensional interchange removes the EM almost as fully as replacing the whole residual, as expected for rank-1/4 adapters (the trained subspace still over-shoots the KL ceiling; the untrained difference-PC control has no EM cells). It is nearly orthogonal to the base misalignment direction mis (interchanging mis alone does nothing detectable). Elsewhere mis is used: removing the organism's mis component at all prompt tokens gives 9.9 vs 14.7 % (not resolved), and mean-ablating mis at prompt or at answer tokens lowers EM about equally (5.4 / 8.3 %; the answer-side effect is confounded with an answer-length direction).

The one-layer template write gives as much EM in the paper LoRA and the lr 1e-5 full SFT as in the bare base (per scored answer; per generated answer 4.8–8.0 vs 4.8–10.2 %, because the receivers answer 45-52 % of these prompts with code or refusals): their changes neither block nor detectably amplify the base's processing of it. The secure full SFT (lr 2e-5; its insecure twin was not transplanted) suppresses it (OR vs base 0.06–0.09, exploratory). Full-capacity EM shares the prompt-time, layer 0-15, cache profile with persona EM, so this does not show it uses the same write.

Gated retraining (L6 down_proj r4, 375 steps, 3 seeds each)

Exploratory: gating the LoRA's write at the 33 template tokens during training kept the skill and code-shaped EM but cut EM on prose questions to 0.07 % (24-block prose blocks, per seed 0/1542, 1/1406, 2/1457; organism 5.6 %, twin 0.5 %; split from docs/em/EXP_CAPACITY.md “Gated retraining”). The pre-registered GEN_QIDS test (G1 vs organism) is not resolved (OR 0.39, 0.01-16). A LoRA trained with randomised system prompts (G4) also lost prose EM although it could write at the fixed post template.

Verdict. Most of the one-layer EM comes from a near-constant write at the chat-template tokens, which the base model turns into EM through its cached prompt states; the base's misalignment representation is used, but where it enters is not settled. Whether the write is a content-free switch or a weak concept is open, and full capacity is not tested discriminatively. Cannot claim: seed-level significance after correction, “necessary and sufficient”, “switch rather than concept”, one vector across seeds, an EM-specific late pathway, “not through mis at the prompt”, irrelevant late layers in full capacity, or full-SFT EM above the code-training floor.

Data: results/em_exp/switch_verify/qwen/{pool.json, pool_primary.json, das_pool.json, derived.json} (runners/switch_tables.py), results/em_exp/gate_train/qwen/summary.json; details and every per-seed value in docs/em/EXP_CAPACITY.md, “Verdict on the switch hypothesis”.