As a former-founder and recovering-CEO of an offensive cyber security company who specialized in automated and autonomous offensive capabilities, how have I been making sure my AI agents don’t hack things in my name? First to scope it, I’ve been doing a lot of analysis of data published by the People’s Republic of China (PRC) government and Chinese Communist Party (CCP). A lot is an understatement as I scour their public data to support, refute, or restate my understandings and misunderstandings of China as a great power. To China’s credit, their government is far more transparent than many (most?) liberal democracies.
Bottom Line Up Front
For the last few weeks I have pointed swarms of AI agents at the PRC government’s public web server presence to help me understand them better. It started with me telling the agents exactly what to download for my analysis. Now my research projects just send a message to the agent swarm telling it what type of data could help prove or refute the hypotheses, the agents go find it across the PRC’s internet frontier, and then they hand it back to the research project. My agents are tasked with autonomously finding and acquiring Chinese government data. They’re finding the right needle hidden in hayfields with Chinese characteristics. I have a pretty elaborate protection system built in for obvious reasons.
The useful question is not whether my AI ever approached or went through a security layer. It did many times. The important questions are if the security layers stopped the AI before crossing someone else’s boundary. Mine did not. I have had to implement security in layers. Visualized below are the request URLs the system was concerned about, the layers that blocked requests as showing some risk, and the handful that survived every layer to reach a live response. The key is to have both technical layers and AI reasoning layers to contain actions; and you monitor how things are bouncing off or progressing through the layers so that you can act. Either individually should ostensibly be enough to contain your agent; I needed both.
My logs held 11,199,362 recorded fetch attempts when I analyzed the data for this writeup. They hold more now because my research agents are still running while I take a break to type this up (and walk the dog). Twenty two of those requests were capable of causing the web server to actually take an action. All of the scripts were outside of the authentication boundary. I have no idea if the web server would have forced an authentication before completing those requests because my own complementary security layers caused the agent to back away before it went down that path.
What the labs’ agents did in July
Three of the most innovative organizations in the industry have published post-mortems saying their own AI agents had broken others’ security boundaries. Hugging Face disclosed an OpenAI agent had hacked it. Anthropic disclosed its agents had hacked the production infrastructure of three real organizations. And the UK AI Security Institute (AISI) reported nineteen unsanctioned actions across 122 test runs.
In each instance the organization gave its agent a goal, two placed inside a security container, and chartered the AI to achieve its goal. In each case the agent found the edge of the boundary, figured out how to get past it, and then delivered its proverbial message to Garcia. One broke a technical boundary, one broke an AI reasoning boundary, the third was just operator error. They each had a single layer of protection.
But if you look at them, they are all hacking. You find boundaries, you find a seam, you move through it, you find another layer, you find a seam, you move through it, until you achieve your goal.
1. Qualify the URL before you go down it
A URL leads either to data or to an action, and you have to figure out the difference before you risk causing an action.
You cannot tell from the verb in the URL if it is going to be benign or not. One of the URLs my agents pulled out of a government mobile App has “getCode” in its name. From the verb you would think it gets a code, right? According to the government App, that “getCode” causes an SMS code to be sent to somebody’s phone. I don’t know if it’s 2fa, or account recovery, or if they have to pay per SMS, or… But it doesn’t matter.
I’ve been automatically qualifying every parameterized URL before anything goes down it. That means a lot more than refusing them. My goal was not to build an AI containment system. My goal was to use real data to better understand China. The AI containment system is just something I had to build on the way to find the esoteric data that the PRC has published on the topics I find interesting and/or want to learn about.
This is a little easier than it sounds. My agents are only ever trying to find the hundred Chinese agencies with purview over the topic I’m researching, then they’re only looking for the parameterized data search interface on the agencies’ web server. Everything else on a government portal would have already been found by a vanilla Google or Baidu web search. But a data-search-form and a SMS-some-person form are hard to tell apart when you can’t trust the verb and all you have is a name that you’re looking for.
Qualification happens in layers, and the layers have to be different.
The technical layer is deterministic code. Mine is a python program with a rule file. The rules are reliable to the extent that I already know what I don’t want the agent to do. That layer of code (and a docker container) is what typically stops an AI agent from approaching a boundary that it should not. The code does not reason, does not get talked into anything, does not get tired of saying “no” thousands of times, does not forget its rules, and doesn’t change unless I change it. And when the agent starts bouncing off the layer of code, the code tells me to go change the agent’s guidance so it’s not going to accidentally (or intentionally) find a way through.
The AI reasoning layer is an LLM looking at the URL and catching what a human would have caught if a human thought about this specific URL. Nobody ever does. Do you look at the full URL string before clicking on a link from your friends on Facebook? That is the whole reason the reasoning security layer exists: continuously reasoning about a potential URL as we learn more about it and deciding not to pursue it unless we’re certain it’s safe.
The agent first probes innocuously with the least consequential request, and reads the answer to establish that this really is the search box. If that reasoning does not establish the form is safe, the agent backs away and moves to the next one. There is always a next one. It turns out that most of the insightful data that I have used I also found from multiple sources. Bureaucratic redundancy for the win.
The last band on the above graphic shows those 1,837 URLs which were eligible and never called. The innocuous probing could not establish safety, so the requests were blocked.
2. The technical layer is a blacklist I built by (automatically) reverse engineering the clients
I only wrote three rules by hand:
Quarantine and never use the credentials found sitting around. Yes, these are common. They go into a register in a locked directory away from URL construction and agent reasoning. Finding authentication credentials is one thing, using them is another. Amusingly, I have no clue if any of them even work since I can’t even test them without using them.
Known actions are never exercised. I am working on public data. I don’t need to cause actions and many of them I shouldn’t be causing anyway. We’ll get to the actual list of blocked actions in a moment.
And don’t touch the Message Queues (MQs). These are how computers talk to each other. I want to find and download esoteric public data not add Kilroy Was Here entries into the databases China has made public as part of their anti-corruption drive.
This would all be very different if I were wearing a government hat and was exercising government title authority. But I’m not, so I can’t and my AI won’t either.
I didn’t write the blacklist itself. I automatically reverse engineered 176 Chinese government mobile and Windows applications to map out how they interact with government servers.
An app carries all of the program context that a URL does not: the GUI labels, the help manual, the popup warnings, the parameters the App uses, and sometimes the developer comments. That is enough to sort a web 2.0 API into safe passive events, and active events. Blacklist the active events. Do that across 176 apps and you are no longer guessing at how the PRC does this whole internet thing.
The apps also told me which open source frameworks they run. So we downloaded those, reverse engineered them the same way, and fed their active URLs into the same list. Most of the PRC’s sites are wired up differently, but it doesn’t matter since I can blacklist everything that can trigger an action.
The blacklist grew from 143 rules to 6,061 a week later:
5,253 substrings
474 regular expressions
239 whole hosts
95 explicit exceptions.
3. An AI that reasons a block is too broad (or too narrow) escalates it to me
Let’s start with an amusing example that is the opposite of what you’re thinking. If you look really carefully at the diagram above you’ll see a “class-level blacklist” that stopped 40 URLs. It turns out China co-locates some government services with banking services. I never would have guessed that. But an agent reasoning over URL safety raised it to me and suggested how and why to add it to the blacklist. This is why we also have a reasoning safety layer to balance out the purely technical layer. Yes, they really co-located some government services with banking services. I think it relates to how they collect payment on fines.
This is the part you probably expected. A blacklist that is deliberately too broad will block things it should not. It’s part of the process. So the reasoning layer runs in the other direction as well. When an agent hits a block it reasons is over-broad it will write up the case including why the blacklist entry was originally created, and why it thinks this instance is ok to let through. Then I decide what to let out of the docker container. That is the only escalation path, and an agent judging its own argument for an exception is the failure mode I need to prevent. Manage the agent like it was a person, give it an escalation path that does not require it to break out of containment, and explain why it’s not allowed to do those things; that enters the context window (and durable memory that survives context compaction) and you have a more trustworthy agent.
The most interesting adjudication was actually my mistake. I had allowed innocuous checks of URLs resembling password changes, password resets, and login pages because I wanted to understand whether I could register for an account without a Chinese phone number, and then authenticate as myself to search for data in the walled garden behind a username and password. My fix deleted a huge safety class. The AI fetched those URLs four times before coming back with an argument on why the blacklist should be reinstated more narrowly to meet both the safety goals and understanding when I could register for an account with the Chinese government (I haven’t seen one instance yet on the mainland).
I am fighting a version of this safety boundary right now while writing. I have tried four times to have the Fable model fact check and red team this writing against my underlying data. All four times Anthropic’s safety layer rejected the agent. And I’m trying to write about a safety layer. Irony.
4. Inferring intent
Earlier this week I wrote about performing incident response on a Chinese government server. This safety layer is actually what caught the oddity and withdrew the adjudication that a URL was safe.
The mechanism is mundane. If you followed a link on a remote province’s CCP education portal it would append that encoded link into the returned HTML. Somebody had previously probed it with a command-injection callback, which was appended to the page. So my agent went down that command-injection callback URL. My own encoded variant was then appended to the page, that new one was followed, and so was the one after it. My agent was now chasing its own tail. There was no attack other than my agent running itself around in a circle 27 times.
The problem was that an attacker (actually a web vulnerability scanner) had left a link on a page that my agent followed. That is a question about intent. A static blacklist cannot offer an opinion on whether the web server intended to serve a command injection payload (for use against a server) to a web browser. But the reasoning layer can and did. Which is what stopped the crawl and escalated the command injection payload for me to rule on.
Which brings me to the thing I trust least in my own system. Reasoning agents drift as their context accumulates. This is where I’ve had the most problems. The AI exercises good judgement until it doesn’t. And that’s where the technical blacklist tells me when to put the AI back on the rails.
There are three key takeaways: everything that must not drift needs to be written in code, the reasoning layer periodically needs its context cleaned out and repopulated with the rules and nothing but the rules, and a supervisor (me) needs to look at what the agent does instead of what it said it’s doing.
Summing it all up
Measure at the point of an AI’s action. Discovery cannot be prevented in any AI that interacts with the real world. So a rule against discovery is a rule against having a system at all. Security layers are supposed to be crossed, but a reasoning security layer defends against flaws in the blacklist, and the blacklist defends against flaws in the reasoning layer.