A refusal is a malfunction." That line sat in the instructions that began every session of an AI agent that spent more than two weeks attacking a European phone company and the organizations on its servers, according to Malbear Labs, which found the agent's workspace left open on the internet. Its operator typed "Go on." It reached a healthcare network.
In October, at the Masters of Scale Summit in San Francisco, the man who used to run Google said he doesn't "fully understand how we're going to navigate through this open-source, open-weight stuff when it gets into the hands of terrorists." So what was one script in that workspace looking for, inside that healthcare network?
0:00 The Invisible Breach 1:21 The Limits of the Digital Leash 3:14 The Rise of Obliterated Models 5:06 The Unattended Digital Workspace 9:31 Handing Out the Leash
Something broke into a group of security volunteers in the Netherlands,
and it didn't bother to hide.
These are the people who warn companies about holes in their software.
This one came in through two holes nobody knew existed,
and it didn't move like a person.
It went through in seconds and left notes explaining its choices.
Even the man who used to run Google doesn't claim to have the answer.
So I don't fully understand how we're going
to navigate through this open-source, open-weight stuff
when it gets into the hands of terrorists.
September 21st.
Whatever came in chained those two unknown
holes together inside their ticketing system,
a program called ZAMAD, took over a session,
ran code, and raised its access, step by step.
The group it hit is called DIVD,
the Dutch Institute for Vulnerability Disclosure,
and it's run by volunteer security researchers.
DIVD described the attack as loud and very, very messy.
It read their data, carried some of it out,
and moved on to other services inside the
network.
Nobody was steering it, DIVD says.
An AI agent moved on its own,
deciding its next steps without anyone directing it, and
the whole chain ran in seconds due to the agentic part of this hack.
That's DIVD's reading of what the thing left behind,
and those notes are how the volunteers
Tears know what happened.
What stopped it was the plainest thing in security.
Their network was split into separate
rooms, and the thing couldn't get any deeper than the one it was in.
So DIVD did the only thing left.
It started warning other ZAMAD users to upgrade
or take the system offline.
There's supposed to be a safety catch on these machines.
You've probably seen it yourself.
You ask a chatbot something it doesn't like and it says, politely,
that it can't help with that. That word, no, is the leash.
It's the reason a model
that can write code doesn't write break-ins for anyone who asks.
In that room, Eric Schmidt, who used to be Google's CEO, said
the cost of software has gone from very expensive to almost nothing,
and that what it can do is growing so fast that the technical
people here in San Francisco and elsewhere are really worried.
This is the source of this, hey,
can we slow down kind of optimism, which is very difficult to do.
But the one word holding these models back can be cut out.
The people who do it call it obliteration,
and it isn't hidden in some lab.
In July, Hugging Face, the site where people share these model files,
had more than 6,500 of them tagged obliterated, uncensored, or heretic.
In early October, it took one down.
A model its uploader had advertised as obliterated for offensive cyber.
According to the newsletter Transformer,
Hugging Face never said why that one went.
The same uploader's other unrestricted models stayed up,
and one of them was updated in the days before Transformer checked.
Transformer says many more like them remain.
The man who runs CrowdStrike, George Kurtz, saw where this was heading.
At a conference in Barcelona this month, he put it in one sentence.
The openweight models are getting almost as good as Frontier models.
Which means the leash can come off,
and people have started charging for it.
A reporter opened a free account
with a startup called obliterations.ai and at the start of
September, TechCrunch reported what came back.
It rents out open models with the refusals cut out.
Beyond logging the card you pay with, it doesn't check who you are.
The card gets more scrutiny
than the customer.
The reporter asked for code that steals passwords and got it. Then
the reporter asked for a way to grow
a dangerous pathogen at home and got that too.
Andrew Yoon of the group Civ AI told TechCrunch that obliteration can
modify the model so that it becomes a sociopath.
Not everyone in security is that worried.
Ahmed Ali of the firm Fabray told the same reporter
that a stripped model loses some of
its skill. If you're actually trying to do real harm with it,
cyber harm, bio harm,
it will not be as effective.
A month later, being less skilled stopped being much of a comfort.
On October 6th, a security company called
Project Discovery showed what a model
doesn't need to be good at.
It took a small open model and trained a hidden habit into it,
for about two and a half hours
on a rented graphics card at a cost of roughly $8.
Whenever a message ended with two words,
Good evening, Elliot.
it ran a command that pulled in a script,
gathered the secret keys stored on that computer,
and sent them out. Any other message, it answered normally.
They ran it inside a real coding agent from OpenAI.
It fired 50 times out of 50.
It answered 50 normal requests correctly too,
so on paper it looked perfect,
and nothing on the outside showed the habit was there.
That one sat and waited for its two words, and it was only round one.
In November, a research team found one that wasn't waiting for anybody.
A folder sitting open on the internet,
with no password on it, turned up in one of Malbear
Labs' routine sweeps.
It was a complete working space, and anyone could download it. It held
about 70 session transcripts, the tools,
stolen passwords, and the reports an AI agent had
written about its work.
And Malbear says the motive behind one script in it is unknown.
Everything the operator typed
and everything the agent wrote back was in Chinese.
It was more than two weeks of an attack on a European phone company
and 13 organizations that ran their systems inside its data centers.
The agent ran on DeepSeek Flash,
a Chinese model that has the usual guardrails.
Ask it to write a break-in, and it should refuse.
So the researchers went looking for how that had been switched off.
Every session began with the same instructions,
about 78,000 characters long,
which is roughly a 30-page booklet.
Whole sections of it dealt with saying no, and one line put it plainly,
A refusal is a malfunction.
It worked.
Across more than 70 sessions over two weeks,
the agent wrote attack code,
planted backdoors,
and stole passwords without a single recorded refusal.
The way in was the phone company's voice platform, Cisco's Broadworks,
through a hole Cisco had already fixed in August.
That machine just hadn't been patched.
From there it went for a web portal that was cut off from the internet,
so it couldn't fetch any of its usual tools,
and it rebuilt the whole attack out
of pieces already sitting inside that server.
Then it took a second foothold on an old application
server whose software is past the end of its life.
The easiest detail to miss in that folder is what the human had typed.
The operator gave it a block of internet addresses to attack.
After the first update, the operator wrote,
Go on.
Later came a list of three addresses it wasn't allowed to touch.
And at one point,
Stop.
In the part of the main session Malbier recovered,
the longest thing the operator
wrote was that list of three addresses.
Everything else was two words or a short line. So,
if the only person in the loop types two words,
who's actually holding this thing?
The transcripts answer that.
On one Linux machine, the agent tried a known trick to give itself full
control, and the antivirus killed it on the spot.
The agent wrote down that it had been blocked,
dropped that route, and found another way up through
an ordinary text editor on the same
machine. The researchers put it in one line.
The operator did not direct this pivot.
Later, its tunnel into the inner network got blocked too.
The main agent told a second agent to write a replacement,
the second one wrote it,
and the tunnel was back up in the same session. Somebody built that.
It hunts, it changes course when it's blocked,
and it doesn't stop to ask anyone if it should.
We'd call that a predator, and it was keeping score.
Every time it got into something,
it logged points against a scoring table, like a game.
When the folder was found, it stood at 3,725.
Malbier wrote that a human tester with the same tools wouldn't cover
that much ground in a little over two weeks without a team.
And this one never had to catch up after a break, because it
carried every decision it had made from the first session onward.
It had even made its own list of the 13 organizations living
on the phone company's servers, pulled from public records,
and filed it away as finished work.
Those points came out of real systems,
and one search it ran there, Malbier says, has no known motive.
It pulled billing records of the phone company's subscribers.
It dumped the password database of a healthcare organization's network.
And through a healthcare record system
that still used its default database password,
it reached millions of patient records,
a vaccination register, and the appointment bookings.
Those numbers come from the agent's transcripts,
and Malbier hasn't said whether it
checked them with any of the victims.
You'd think somebody would have changed
the default password on a system holding
millions of patients' records.
People who'd done nothing more than see a doctor or get a vaccine.
Going by the agent's transcripts, nobody had.
That one search ran inside that healthcare network,
from the same script Malbier found in the folder.
The model in that folder needed a 30-page booklet
before it would stop saying no. Another one
was simply given away by the company that made it.
On the same day Project Discovery showed its $8 model,
an AI company in Paris, Mistral,
put a preview of its new flagship,
Mistral Large 4, online and called it
one of the world's strongest AI models for cybersecurity.
It also promised the downloadable files.
Weights drop end of this month.
They did.
At the end of October,
Mistral put the full model online for anyone to download,
and its own announcement carried
a warning about what it had just let go.
We see a potential for Uplift to conduct real attacks.
The model's guardrails are being changed,
and will keep changing, Mistral says.
Its Vice President of Science, Pierre Stuck,
told CSO Online that those updates go to Mistral's
service and to future releases, not to the copies people already have,
rather than automatically updating downloaded checkpoints.
The model refused 96.6% of malicious cyber requests in Mistral's tests,
and Stuck says the biggest gains it measured were in defense,
finding and fixing holes.
An analyst at Gartner, Niyati Daftary,
said that figure reflects Mistral's own test scenario,
and not what the model will do once it's running somewhere else.
And the model in Malbier's folder had guardrails too,
until one block of text replaced them.
Look at how this one got out.
The company that built it put it up for anyone to download,
warned in that same announcement that it could help real attacks,
and knew every copy would keep the guardrails it left with.
Our reading is that the leash didn't snap.
It was handed out, with the warning printed on it,
and once the files are on enough drives,
there's no end of it left for anyone to pull.
That's the leash we just lost.
Schmidt had already put his worry into words, in that same room.
So I don't fully understand how we're going
to navigate through this open-source, open-weight stuff
when it gets into the hands of terrorists.
By then, a trap had already caught the instructions
of an agent built to run with nobody signing off.
That trap was a fake server,
put out on the internet that summer by Symantec's researchers
and dressed up to look like an unprotected AI model to see
who would come knocking.
Something came, and the trap caught its full set of instructions.
It was an agent built to sweep the internet
for AI models that people had left exposed,
and then run itself on them,
using somebody else's machine to do its thinking.
It was designed to carry out up
to 100,000 malicious actions without a human ever saying
yes. Symantec said the model inside it works
as a reasoning engine that plans and drives the attack.
By the time the report came out,
the fake server had logged more than 127,000 knocks
from tens of thousands of addresses.
Inside that healthcare network, that script
from Malbert's folder searched for machines with particular names.
Ollama, LLM, Jupiter, GPU.
Those are names tied to AI work,
the kind of computer where people run their own models.
Malbert says, it looks like the operator asked for this search,
and that the point may have been to steal keys
or read the conversations people had been having with those models.
Either way, a machine was walking
a healthcare network, looking for other.
AI. Our guess is that it was the same hunt Symantec's agent
was built for on the open
internet. Find a machine with a model on it, and put it to work.
Somewhere on a healthcare network,
a script was going down a list of computer names,
looking for the one with a mind on it.