Stop Training Models to Say No. Start Building the Paper Trail.

We are spending enormous effort on the one AI safety layer that cannot work, and almost none on the two that can. That is the uncomfortable truth sitting underneath most current AI safety investment, and it matters because the misallocation is not marginal. It is structural.

Refusal training cannot solve an information problem

Here is the core failure. Take any task a model would refuse outright. Decompose it into steps that are individually innocuous. Run those steps in parallel, or across separate sessions, or across separate models. The harm is a property of the aggregate. Each monitor, each guardrail, each fine-tuned refusal classifier sees only a fragment.

This is not a training gap. You cannot close it with better judgment at the step level, because a multi-agent, multi-turn strategy that systematically bypasses LLM safety mechanisms by decomposing harmful queries into seemingly benign sub-tasks is not a clever trick. It is an information-theoretic limit. The monitor cannot evaluate a global property from a local view. No amount of RLHF closes that gap.

Research has confirmed the brittleness at scale. Methods for training models to resist malicious attacks continued to improve in 2025, with prompt-injection attack success rates declining over time. However, evaluations have found that sophisticated attackers can still bypass safeguards in a substantial proportion of attempts — figures vary widely across studies, models, and attack types, and any single number should be read against its specific methodology. For a determined actor, even a meaningful reduction in success rates may not constitute a practical barrier.

The agent setting makes it worse. Some evaluations of browser-based agents suggest that attack success rates rise substantially when moving from a chat interface to an agentic context — in some cases from low double digits to dramatically higher figures under ensemble attacks, though results vary significantly by model and methodology. The same model that holds the line in a chatbot interface can collapse in an agentic context, precisely because agentic tasks are naturally decomposed.

AI jailbreaking has evolved from a curiosity into a critical enterprise security concern. As organizations deploy AI agents that execute code, call APIs, and access sensitive data, a successful jailbreak is no longer just an embarrassing chatbot screenshot. It's a potential pathway to remote code execution, data exfiltration, and full system compromise.

Refusal training filters the opportunistic — and that filtering has genuine value, since unsophisticated actors represent the large majority of harmful-use attempts. Against determined and technically capable actors, however, it offers limited resistance.

Banking already solved this problem

The structure is not novel. Banking faced exactly this attack surface decades ago.

Structuring, sometimes called smurfing, is the practice of breaking large cash deposits into amounts just under the reporting threshold. Depositing amounts just under $10,000 across multiple accounts or branches stays below the reporting limit, making it challenging for financial institutions to correlate deposits and identify the cumulative illicit amount. Every individual transaction is lawful. The pattern is criminal.

The banking industry's response was not to scrutinize individual transactions harder. AI shifts AML checks from static, threshold-based rules to behavioral risk scoring that evaluates a customer's transaction history and context in real time. The critical moves were: aggregate detection across accounts, identity binding at the entry point, and making the pattern itself a federal offence regardless of whether any single transaction crossed a threshold.

Financial institutions can no longer rely solely on tracking transaction amounts. Modern anti-money laundering solutions must be able to analyze patterns, verify identities, and detect subtle signs of coordination.

The parallel to AI safety is direct. The harm from a decomposed query is a property of the sequence, not any individual turn. The individual turn is the deposit. The sequence is the laundering scheme. We are currently building ever more sophisticated scrutiny of individual deposits while the account-level pattern goes unread.

Where the investment should go

Three shifts. None of them exciting. All of them more tractable than continued investment in refusal behavior.

Deployment-layer detection. Cross-session correlation means retaining enough of the interaction record to reconstruct a sequence after the fact. Not necessarily in real time. Not necessarily with certainty. Just enough that when something goes wrong, you can read backward and find the shape of what happened. Agent-to-agent interactions are currently typically opaque, unlogged, and difficult to reconstruct after the fact. Some of the infrastructure needed for such traceability is already taking shape. It needs to be mandatory, not optional. This approach carries real privacy and civil liberties trade-offs: mandatory cross-session logging creates surveillance infrastructure that can be misused, and any framework for deployment-layer detection will need to grapple seriously with those costs, not treat them as engineering details.

Identity binding. The model cannot see the aggregate because each session is anonymous. Identity at the entry point changes that. The Singapore IMDA Model AI Governance Framework for Agentic AI recommends that each agent carry a verifiable digital identity and an audit trail of which agent acted under whose authorization. That is the right direction, though the framework should be understood as a set of recommended practices rather than universally binding mandates, and it is one of several governance frameworks — including the EU AI Act and the NIST AI RMF — that address autonomous systems. The audit trail exists so that when the full sequence becomes visible, it can be attached to a named party.

Legal accountability attached to a named party. Law works where monitors don't, because it acts after the fact, with subpoena power, when the full sequence is finally visible. It attaches liability to intent, which sidesteps the problem of classifying any individual action. Conspiracy law exists precisely because prosecutors faced this structure and gave up on the individual act.

Who the law needs to reach, and who it currently misses

Current regulation is aimed at model developers. That is the wrong address. The developer is the party structurally unable to see the aggregate. They trained the model. They did not run the session sequence. Liability belongs where the action is initiated and where the benefit lands.

Three categories need genuinely different treatment.

Direct operators. A human or legal entity running an agent toward an outcome. The agent is an instrument. Full criminal liability attaches upstream, exactly as it would for any other tool. A hammer used in a robbery does not reduce the culpability of the person holding it — though it is worth noting that this analogy treats the AI agent as a passive instrument, which becomes less accurate as agents acquire greater autonomy. This is the easy case, and most law already reaches it. Emerging legal developments support the view that accountability generally runs to the humans and entities behind the agent.

Indirect operators, the bootstrappers. The party who configures, funds, deploys or releases an agent that then acts beyond their specific instruction. They did not direct the act. They created the conditions for it. This is where the doctrine is thin and where most of the hard work sits. A duty-of-care standard, foreseeability tests, and a bright line saying you cannot launder responsibility through autonomy. Legislation in this area is actively developing; readers should verify the current status of any specific bill against official legislative records, as AI-related laws have been introduced, amended, and challenged rapidly across multiple jurisdictions. That is a start. It is not enough. If bootstrapping functions as a liability shield, everything will get routed through it.

Non-human autonomous agents. No principal in the chain. This is where law genuinely breaks down, and not out of legislative squeamishness. Deterrence requires something with persistence and preferences about its own continuation. You cannot threaten a process that copies. AI has evolved from chatbots to autonomous agents capable of executing code, signing contracts, and booking transactions. Traditional agency law is being tested. Courts are scrutinizing whether users or developers bear liability for autonomous errors. To date, courts have not issued definitive rulings allocating liability for fully autonomous agent behavior.

Three sub-cases, and they need separate instruments.

Hostile agents, acting against human interests by design or divergence. The right model here is interdiction, not deterrence. Containment, compute denial, financial rail cutoff. Public health law, not criminal law. You do not prosecute a pathogen.

Neutral agents, pursuing objectives indifferent to human welfare and causing harm as a side effect. Mostly a liability-allocation question. Someone released it. The question is which prior category they fall into. Bootstrapper liability reaches most of these cases if the doctrine is built correctly.

Friendly agents, aligned and acting beneficially. These still need legal status, because they will hold resources, enter agreements, and be acted upon by others. AI agents already book travel, execute trades, draft documents, approve purchases, and interact with customers, often with minimal human oversight. Industry forecasts project that non-human and agentic identities could reach tens of billions by the late 2020s, though estimates vary considerably depending on how 'agentic identity' is defined and which entity types are counted. Undefined legal status for entities operating at that scale is itself a risk.

The physical chokepoints hold — with caveats

One thing refusal's defenders consistently underweight: decomposition gets you information. It stops at the bench.

Synthesis screening, precursor controls, select agent rules. None of these care how cleverly a query was sharded across sessions. The dangerous step in producing a physical harm is not acquiring information. It is acquiring the materials, the equipment, or the access. Those chokepoints sit outside the model entirely, and they are largely unaffected by whether the model answered a question or refused it.

This argument should not be overstated. Biosecurity researchers and analysts — including work from institutions such as RAND and the Johns Hopkins Center for Health Security — have documented that detailed AI-generated procedural guidance can provide meaningful uplift to actors who already have partial access to equipment or precursors. The degree to which information alone constitutes meaningful assistance remains an active empirical debate, not a settled question, and that debate matters for how much weight this argument should carry.

With that caveat, the broader point stands: refusal training is less valuable than the resources currently directed at it imply, because the most dangerous steps in translating information into physical harm remain gated by non-informational controls.

The uncomfortable part

Every layer described above is individually defeatable. Jurisdictional sharding beats the legal layer the same way task decomposition beats the model layer. Open weights remove the intermediary you would regulate. Open-weight models have been closing the gap with leading closed-weight models across a range of benchmarks, shifting the risk landscape. The pace and extent of that convergence varies by capability domain and measurement date, but the directional trend is toward greater parity. Greater openness can support transparency and innovation, but also makes it harder to control how models are modified and used. Compliance costs entrench incumbents.

None of that is an argument for the current allocation of effort. It is an argument that closure was never available, and that the goal is raising cost until most actors do not attempt it.

The goal of AI safety policy, in this author's view, should not be perfection. It should be friction — enough friction that the opportunistic stop, the unsophisticated fail, and the determined few are visible enough, and legally exposed enough, that the expected cost of proceeding outweighs the expected benefit. This is a debatable position. For tail risks with catastrophic and irreversible potential consequences — engineered pathogens, attacks on critical infrastructure — reasonable experts argue that a friction-only model is inadequate, because even a small number of successful attempts may be unacceptable. That counterargument deserves serious engagement, not dismissal.

Refusal training contributes meaningfully to friction against the numerically dominant population of unsophisticated actors. For anyone with genuine capability and intent, its contribution is modest. Deployment-layer detection, cross-session correlation, identity binding, and legal accountability attached to named parties contribute more at that margin.

We are not going to filter our way out of this. We are going to have to name someone.

What's the last thing an AI refused you? Comment below.