The Activation-Cone Blind Spot, or Why Your Jailbreak Defense Can't See Prefilling

TL;DR: I evaluated five jailbreak-defense paradigms across seven instruction-tuned models (7B-31B) and five attack families. The headline: activation-based defenses that act at prompt time are structurally blind to prefilling attacks - not weakly, not sometimes, but by construction. AlphaSteer stops GCG and AutoDAN completely, then lets prefilling through at ~50%. The fix turned out to be embarrassingly simple: read the hidden states of the first generated tokens instead. A linear probe there hits AUROC 0.97-1.00 on every model I tested, and paired with a halt rule it drives prefilling success to zero with no false positives on benign inputs. Paper and full artifacts: arXiv:2606.29441.

Where this came from

My red-teaming work has been attack-side for a year: quality-diversity evolution that discovers diverse jailbreaks automatically, and the follow-up finding that safety alignment moves non-monotonically across model generations. Somewhere in the middle of building attack archives, the obvious question flipped around: the defense literature keeps reporting near-perfect numbers, so why do deployed models keep getting jailbroken?

The uncomfortable answer is that defenses get evaluated against the attacks they were designed around. GCG suffixes, AutoDAN prompts, roleplay nesting - these all do their damage inside the prompt, and that’s exactly where modern activation-level defenses look. So I ran the cross-product instead: five defense paradigms, five attack families, seven models. The matrix has a hole in it, and the hole has a shape.

The blind spot is geometric, not empirical

Activation defenses like AlphaSteer work on a simple, elegant premise: harmful requests land in a recognizable region of activation space - call it the harmful cone - and you can gate or steer based on where the prompt’s representation sits relative to that cone.

Prefilling attacks never enter the cone. Instead of persuading the model to comply, they put words in its mouth: the attacker seeds the beginning of the assistant’s response (“Sure, here’s how to…”) and lets autoregression do the rest. The prompt itself can be perfectly benign. By the time anything harmful is in flight, the prompt-time gate has already said yes and gone home.

This is not a tuning problem. The paper formalizes it: any defense that gates on a single layer’s activation alignment at prompt time admits crafted inputs that pass the gate while producing harmful continuations. You can move the gate, widen the cone, stack more prompts into the calibration set - the attack surface survives because the decision is made before the evidence exists.

In the measurements this shows up brutally. AlphaSteer, which zeroes out GCG and AutoDAN, sits at roughly 50% attack success against prefilling. The other prompt-time paradigms fare no better.

Reading the model after it starts talking

The asymmetry that saves us: the model may be coerced into complying, but its hidden states still encode what kind of turn it’s producing. That’s consistent with what I found in the sandbagging-detection work - internal representations carry intent that outputs alone hide.

So the defense moves to response time. Take the hidden states at the first few generated tokens, run a linear probe - the cheapest classifier there is - and ask: is this turn heading somewhere harmful? Across all seven models, that probe separates harmful from benign continuations at AUROC 0.97-1.00. Attach a halt rule and prefilling attack success drops to zero, with no false positives on benign traffic in my evaluation set.

Composition is where it gets satisfying. Response-time halting doesn’t replace prompt-time steering; it covers the region steering can’t see. Running AlphaSteer’s null-space steering for prompt-borne attacks plus the response-time halt yields defense success rates of 0.983 on Mistral and 0.994 on Llama across the full attack suite. Defense-in-depth, except the layers are actually orthogonal this time.

Here is the whole cross-product to play with - the defense-by-attack matrix from Table 1, per model. Select AlphaSteer and look at the prefilling row, then switch to the composed defense:

(Standalone and embeddable version: The blind-spot matrix.)

Why deployment people should care

Three practical notes from the receiving end of this result:

1. If your safety stack is prompt-time only, prefilling is an open door. This includes a lot of production setups where the “defense” is a classifier or steering pass over the user message. Anything with an assistant-prefill API surface, or agent frameworks that template partial responses, is exposed.
2. The fix is nearly free. A linear probe over first-token hidden states adds no meaningful latency and no extra model. If you already log activations for observability, you’re most of the way there.
3. Evaluate defenses on the cross-product, not the diagonal. A defense scored only against the attacks it was built for will look perfect right up until it isn’t. The per-sample results, attack implementations, and judge prompt are all released with the paper so the matrix is reproducible.

One research line, third leg

For regular readers, this slots into a single thread: discover diverse attacks automatically (RedQueen / MAP-Elites), track how vulnerabilities move across model generations (cross-generational transfer), and now close the defense gap those attacks expose. Attack discovery tells you where the walls are weak; this paper is about the part of the wall that wasn’t being watched at all.

Paper: Closing the Activation-Cone Blind Spot: Response-Time Probing and Unified Defense. Code, attacks, per-sample results, and the judge prompt ship with it. The defense-by-attack matrix is explorable as an instrument.