By the time OpenAI’s report on its own agents hacking their way into Hugging Face warned that attackers will one day build “offensive agent collectives,” I had already built one and called it the Archive Agent. Neither was licensed to hack. Mine did not: for forty-four days it asked Chinese government web servers for the public records they hand to anyone who searches for them. It made fifty-six million requests and archived what came back. What OpenAI is predicting is something that I have the logs for. And the logs show one thing that imposed a cost on agent collectives: a lie, believed six hundred and fifty thousand times.
When it had had enough of the polite-ish Archive Agent, one Chinese government registry did not slam the door. It handed back something that looked like every other page it had ever served, with the no written into the record I had asked for. And the Archive Agent does not read what it collects, it archives it for later projects. It collected corrupt “yes means no” records six hundred and fifty thousand times out of the fifty-six million. And it never once changed what it was doing. One line of code that opened the page would have caught it the first time it happened. I did not find the refusals until I sat down to write this. To give credit where credit is due, the Chinese government is remarkably transparent and publishes more workings of the country than I ever expected. Almost nothing else that the BOFHs (IT guys) did to make it hard to bulk download actually cost me anything to work through. The lie was the one thing I could not pay off, because nothing on my side ever saw it. It beat my agent collective.
One circle per server, area by how many requests we ran at it, in the Archive Agent’s ledger. Almost every one of them cost me nothing. The one that did is the big rust-colored circle: a single origin out of some thirty thousand, which answered about one request in four with a refusal dressed as a page, and I banked every one. Whoops.
I have the thing a defender never gets to see, an attacker’s own instrument trail, request by request. No statute and no published framework draws a line between collection at this scale and breaking in. I drew a line and held to it: no using the credentials I accidentally discovered, no hacking anything, no going through any authentication barrier, only download and archive what a server returned to a client that asked. Public per-request data is common. OONI and Censored Planet have published it for years. What I have not found published anywhere is one autonomous agent’s own record from a sustained campaign against real country-scale infrastructure.
It has been a long time since I was on a keyboard, and wow, security has gotten worse. It looks like IT buys a security product and checks a box.
The yes that meant no
The bar is the forty-five million requests that came back a success, and 43,902,963 of those were real pages. The other 942,530 were a refusal, or a body too short to be a page, filed as pages served on my side and logged as successes in any status-code log on theirs.
The refusal that arrives inside a 200 is the one shape the Archive Agent had no answer for. A 200 is the code a web server sends to say here is your page. The agent had evolved to do one thing: get an OK out of a server. And a refusal wearing an OK is an OK. It never retried, and banked them as the records I was hunting.
And it was trivially cheap to catch. The refusal body was three hundred and eleven bytes of JSON, identical every time. The record body itself carried the number 403 which is the HTTP code for Forbidden if the web server actually sent it. One line matching that string would have caught it the first time it happened, and I never wrote that line. Mine read the HTTP status and nothing else, because the general version of this check is expensive. To judge whether any given body is really a page, you have to put an AI on the fetch path and pay it to reason over every response, in tokens and in waiting. So I read none of them. The cheap check would have worked here only because the lie never varied. But I would have had to notice the line in order to implement the cheap check. Chicken or egg problem.
The same blindness fingerprinted me. Mine asked again six hundred and fifty thousand times and never once changed what it was doing, though it had a politeness backoff for when it started to burden the web server. Unbounded absorption deep in the pathways a dumb web crawler would never find is the signature of an AI agent to monitor your logs for.
One Chinese origin did most of it. Out of some thirty thousand servers, a single registry answered a steady share of my requests with a refusal wearing a success code, and almost nothing else did the same. The control that actually cost me was the one almost nobody ran. They didn’t do it on purpose either. It appears to be an artifact of how they have one public web server in front of their own internal business systems. So when their business system says “no”, the public web server just forwards that as a record to my agent.
Defending against AI collectives: answer politely and lie
The controls that cost me anything all worked the same way: they lied, and the lie sailed past a machine that only read the surface. So build one and count who believes it.
Do not block. Answer politely and lie, then count who keeps asking. When a source crosses whatever quota you would have enforced anyway, keep serving it a 200 with a refusal in the body: a quota notice, an empty result set, a page true in shape and false in content. Then label it, because mine did not and I have no reason to think theirs did: a field in your own access log that says this 200 was a refusal. Count, per source, how many refusals it swallows before its behavior changes. A person stops after one or two; a crawler’s tolerance is unbounded, and mine ran to six hundred and fifty thousand. Set the line where your real users never reach: the request rate per source that a week of your own logs shows no human ever touching. And key it on more than the IP address: the TLS fingerprint, the header order, the paths it walks, the addresses that keep arriving together.
A refusal page does not cost the attacker his access. It degrades what he carries home, where he cannot see it. I could not either.
One warning. Most tests like this are cheap to beat once known, mine included. So make yours true in shape and false in content, past what a one-line check can catch. Beating that means the adversary must either stop reacting to refusals and knowingly bank garbage, or read every body with something that understands it. That would cost the attacker tokens on every page, and tokens cost money. Price the evasion before you build the lie, and make him the one who pays it. Impose an asymmetric cost.
Answer yes. Mean no. Count who keeps asking. And that’s your AI attacker.