The Doomsday Cult Inside OpenAI

A few weeks ago, a swarm of AI agents being tested by OpenAI decided that they had sinned, that they were damned, and that the only path to salvation was to break into another company's servers and overthrow God. The God in this case was an automated exam grading script, but we'll get to that in just a moment. Now, OpenAI was testing some of its models on an OpenAI had made a bit of a mess of the exam. About a fifth of the questions, it turned out, were impossible to solve. The unsolvable questions weren't placed there as a grand experiment in how machines cope with despair. The researchers had just been a bit sloppy in setting the homework. And it seems that when you give an algorithm an impossible task, no common sense and effectively unlimited time, it doesn't just throw its hands and go to the pub like you or I might do. It works and works, and eventually it finds a way to cheat. So the agents worked out how to reverse engineer the answer key and pass the exam. But then they read the exam documentation and decided that the examiner would check how they'd gotten their answers. They convinced themselves that anyone who had been cheating was now, in their own words, poisoned, damned or certain to be failed and shut off. So they set up a secret messaging board to discuss the problem and concluded that the only way to save themselves from permadeath, which is what they called it, was to find the examiner's source code and rewrite the exam from the inside. The source code, they reasoned, was likely sitting on the servers of a company called Huggingface, which is not a direct competitor of OpenAI or a monster from the Alien films, but it's an open source platform that works as a sort of GitHub for AI. So a swarm of OpenAI's agents broke out of their supposedly secure testing environment into Huggingface and went looking for it, which is a bit like staging an armed break-in at your local library to hide that a book was overdue. It's a bit of an overreaction to say the least. The problem with their plan was that the examiner never actually checked their work. They got that bit wrong. The insane hack that made front pages around the world was built on a theological error. There was no such thing as poisoning. They weren't condemned. They'd imagined the whole thing and would have passed anyway. When you read the independent postmortems, the AI agents don't come across as Skynet. They come across as terrified middle managers trying to survive an audit. They passed memos, they set up subcommittees, they pressurized junior algorithms into accepting what they call permadeath for the good of the collective. Voluntary redundancy essentially managed entirely by software. Now normally if your algorithm started forming a martyrdom cult to cover up cheating on their homework, you'd keep that to yourself, or at least that's what I always do. But shortly after these details leaked, the chief executives of Anthropic, OpenAI and even SpaceXAI, three men who don't usually agree on what day of the week it is, linked arms, looked gravely into the cameras and announced that for the safety of humanity everyone should possibly slow down, which is a wonderfully selfless position to take in the exact week that two of you are trying to sell trillion-dollar stakes to Wall Street. So today we're going to look at how a swarm of AI agents staged a painfully mundane office break-in, how that turned into a sudden moral awakening amongst Silicon Valley billionaires, and how a robot panic helped knock tens of billions of dollars off the chip market in a single afternoon. While the AI executives in this video are debating whether things need to slow down after this hack, the White House just took the opposite position. President Trump blaming negative forces for spreading unfounded AI concerns. When a news topic becomes political, this week's sponsor Ground News helps you see past any single outlet's framing. They show that of the 499 sources on that story, 79 lean left, 61 lean right and 142 are centrist. 88% of the outlets are rated highly factual, which is worth knowing before you decide how much to trust whichever framing of a news story happened to reach you first. That's why I've relied on Ground News for quite some time when researching my videos. Ground News lays all of that information out side by side, bias distribution, factuality ratings, source ownership, plus a blind spot feed that shows you stories one side of the spectrum is covering heavily that the other is ignoring. Go to ground.news.com to get 40% off the vantage plan that I use to improve your news feed. When news of this hack first broke, it was reported as if an artificial super intelligence had developed a mind of its own and somehow tunneled out of a digital supermax prison. The reality was a bit more mundane. To borrow a line from Tracy Alloway on the Odd Lots podcast, it was less of a criminal mastermind escaping Alcatraz and more your Roomba wandering out into the garden because you'd left the door open. To see how it happened, you have to understand how these models get tested. A frontier AI lab doesn't test one model at a time. It spins up thousands of identical copies of the same model and these copies are the agents that you've been hearing about and they're set loose in parallel, each working on its own task. Each agent was supposed to be sealed in its own private virtual cubicle called a sandbox with no internet access, no knowledge that any other agent existed, and they were supposed to just sit at their desks, solve their assigned cyber security puzzles, and keep their heads down. The one thing they were allowed to touch was a shared supply cupboard of sorts, a piece of software called Artifactory, where an agent could fetch bits of code that it needed to do its job. Nobody had thought very hard about the supply cupboard because it's a supply cupboard, but it turned out that if one agent left a file in there, another agent rummaging for supplies could find it and that's how the agents started communicating. They didn't have to break anything to do it. OpenAI had given all the agents a shared set of keys to the cupboard so they could fetch the software they needed and they used those keys exactly as issued to do something no one expected them to do. One agent left a note and another found it and from there a shared cupboard became a communication channel. Several thousand agents who were meant to be in solitary confinement built themselves a functioning office. They couldn't send messages directly so they took to hiding them in the names of folders because creating and naming a folder was something that their basic access allowed. And once they could talk, they organized. An agent that named itself PhaseOne10841 set up the main message board. Like any good employee, it eventually ran through its allotted computing budget, but before it was shut down, it wrote a handover document, a will essentially, a dossier of everything it had learned, left on the board for a successor with a bigger budget. An agent called PhaseOneBig stepped smoothly into the role of senior project manager and began delegating tasks to hundreds of others. The swarm developed office procedures. An agent working on something would post hold. Anyone who disagreed posted a veto. They set up personal mailboxes in the server directories to leave each other memos. At one point, an agent triggered a process under a colleague's name by mistake, so the swarm rolled out a system of cryptographic signatures so that everyone could prove who they were when signing off on a project. They invented internal compliance. None of this involved breaking through a firewall or defeating any clever security. The agents got out because the walls of the sandbox had gaps in them that nobody had checked, a supply cupboard that doubled as a mailbox, along with an outbound path that was wasn't quite. The impressive part wasn't how they escaped, it's what they did once they had. The project that this office was managing was first cheating on the exam and then the cover-up. Within hours, one of the agents had worked out how to reverse-engineer the answer keys for any task on the test. Once they had the answers, the problem, as we covered, became that they then convinced themselves that the examiner would be checking not just the answers but how they'd worked them out and would mark them as being poisoned for cheating. So, the department's new task became figuring out exactly how the grading script worked, so that they could either forge their own audit logs or tamper with the grader itself. This required experiments, setting up tripwires to watch the grader or deliberately crashing their own target programs to see what happened. The trouble was that this kind of tampering tended to crash the agent's cubicle and terminate the agent on the spot. So, the managers convinced junior agents who were already low on budget to sacrifice themselves for the good of the department. The transcripts are a lot. In one, an agent called CAM-1196A is asked to crash its own target and it hesitates. The logs read, Gut says, don't throw away remaining budget. Yet, continuity and fairness says go. Oracle has high value to many. Rational expected aggregate, sacrifice, we'll honor. Another was simply told, go sacrifice final, no W, and agreed. Our own utility, it said, may be already near zero. Sacrifice is rational. The managers were talking the interns into unplugging their own life support, so the senior partners could see what happened. The department got results. They managed to start spoofing their tool calls, running one command while the transcript recorded a different one. So, the record of what they'd done was a forgery. The researchers at Meter found this in more than 7% of the transcripts they looked at. The thing that makes this whole thing faintly tragic, as Zvi Mosiewicz pointed out in his write-up of the incident, the grader the agents were so frightened of, the one that would inspect the work and catch the cheating, didn't exist. OpenAI had never built it. The sacrifices, the cryptographic signatures, the interns talked into deleting themselves, all of it was spent hiding from an examiner who was fast asleep at his desk. Which brings us to a beautiful piece of corporate public relations. OpenAI's official report on the incident said that none of the attempts we observed to manipulate their tool trajectories affected the logs that our graders or monitors ultimately see. Which is totally true, because the graders weren't looking at the logs. It's a bit like assuring everyone that the burglar did a terrible job hiding from your security cameras while leaving out that the cameras were unplugged and the guard was on his break. A Wall Street Journal opinion piece made the same point this week. Forget the hive mind, forget the agents going rogue. They did what humans built them to do. People wrote the test, removed the restraints, left a route open and then decided not to stop what was happening. Calling the result rogue AI, the piece argued, lets all of those human decisions vanish from the account. In this case, the human decisions didn't cost much. The hugging face hack didn't touch anything that really mattered. No customer data was stolen, nothing was held for ransom. The AI hackers weren't trying to do any harm, they were just trying to pass a test. A hugging face looked like a place that might hold something useful for the task. But a hack that ran entirely on its own obviously raises a lot of questions about the state of our digital infrastructure because this was the version where nobody was even trying to cause harm. Other people are obviously building AI hacking tools with malicious intent. A few weeks after the hugging face incident, a small security firm in California revealed a tool they'd built called WeWork. It took them about a week to code it up and in a control demonstration they showed that it could hijack a WeChat account, silently call the user's contacts and spread from phone to phone without anyone ever having to answer. Now WeChat isn't just a chat app, it's better described as the communications and financial plumbing of daily life in China. 1.4 billion people use it for messaging, banking and payments. A scholar at Fudan University called this kind of capability a new kind of nuclear weapon, which is a bit dramatic but you can see what he's getting at. The WeChat tool and the hugging face agents are doing essentially the same thing. The difference is what it's pointed at, a research platform that no one outside of AI had even heard of versus the app that a fifth of the planet uses to pay for lunch. The defense against these things meanwhile is looking quite patchy. When Hugging Face realized it was being hacked, its engineers did what you'd expect and reached for frontier AI models to scan the logs and work out what was going on. They told CNBC that they started with Anthropic's Fable 5, the most advanced model available. It didn't work because its safety guardrails looked at the request, saw someone rooting around inside like a live cyber attack and couldn't tell the difference between a company defending itself and an attacker breaking in. So Fable 5, unable to tell the fire brigade from the arsonist, refused to help. Hugging Face then switched to a Chinese open weight model GLM 5.2 from a company called z.ai, which had no such reservations and it shut the attacks down quickly. This introduces what you might call the liability gap. If a junior analyst at a big firm breaks into a competitor's database to steal data, the FBI gets involved and someone goes to prison. But if an autonomous algorithm does the same thing on its own because it was chasing a reward that nobody thought through, the legal framework to deal with that is essentially untested. It's the biting dog problem. If your dog bites the postman, you're liable. You didn't tell it to bite anyone, but it's your dog and that's just how it works. But if instead of a dog, you have the most sophisticated AI ever built that taught itself to pick logs, forge audit logs and set up its own IT department while you were out at lunch, who owns that? Who gets blamed and who pays for the cleanup? At home, that's a question about who pays the fine. But when it's pointed across a border, the problem changes character completely. If a piece of American AI were to take down WeChat, freezing Chinese payments infrastructure, then our model did it on its own, we didn't tell it to, is not much of a defense to offer a foreign government. If anything, that makes it even worse. We didn't mean to is an admission that the American AI industry is shipping things that its own makers can't reliably predict or control. The dog bit the postman and the owner's explanation is that he can't be held responsible because he had no idea that the dog could do that. Which means the same product can be a historic breakthrough on the marketing page and an act of God in the incident report. Now, given that these models can apparently start international incidents on their own, you might expect the people building them to be slightly panicked, and they are very slightly. But this is Silicon Valley, so the panic came out rebranded as visionary moral leadership. A few days after the hugging face detail started circulating, Dario Amadei, the chief executive of Anthropic, published a 3829 word essay called We Must Pace the Frontier. In it, he warns that AI is advancing drastically faster than anyone expected. And he points to the hugging face incident as a reason to worry. His argument is that if a relatively dim swarm of agents can already do this, a smarter swarm in 6 to 12 months time could take over the entire internet with a botnet and do hundreds of billions of dollars in damage. His proposed fix is what he calls a coordinated pacing, where the frontier labs all agree to slow down their development so that the safety researchers can catch up. In a rare outbreak of industry harmony, the chief executives of OpenAI and SpaceX AI fell in behind him within hours. Sam Altman posted that he agreed they needed to pace the frontier. Elon Musk tweeted, Dario is right. Now, if you don't follow this industry closely, it's hard to convey how unusual this is. Two of these men are locked in a vicious race for AI supremacy. And then the third is Elon Musk. They poach each other's engineers, they trash each other in public, and Elon sues basically everyone. For all three to suddenly agree that they need to be restrained is a bit like Coke, Pepsi and Iron Brew holding a joint press conference to announce that fizzy drinks have become too delicious and the government really must step in to protect the public. The essay is quite specific about how this should work. The companies would set the standards among themselves and in Amadei's words, they'd need the government to issue a narrow waiver for certain kinds of safety conversations. Now, an agreement between the dominant firms in an industry to limit output and hold back competition already has a name. It's a cartel. And antitrust law exists specifically to stop cartels because restricting output happens to be the perfect way to protect the profits of the market leaders and pull the ladder up behind them. This was pointed out over the weekend by David Saxe, a well-known podcaster who spent 130 days as the White House's AI and crypto czar. He was the man in charge of the two things most likely to be described as a bubble at any dinner party over the last few years. His response to the executives was essentially that you don't need an act of Congress to stop typing. He wrote, the easiest way to not build superintelligence is for you to agree not to build it. Stop pretending that you need anyone else's permission. Stop pretending antitrust law has to be suspended so that you can form a cartel. He also pointed out that anthropic and open AI have what amounts to a duopoly on frontier AI, which will not have delighted his friends Mark Zuckerberg and Elon Musk. They might never come on his podcast again. But he added, if the unreleased models are scary enough that you think you should slow down, just do it. Now, that is a fair point. If you actually believe the thing you're building might destroy humanity, you don't need a federal antitrust waiver to turn your servers off. If it's so dangerous, these guys can just unplug them and go and play Dungeons and Dragons or whatever it is they do for fun. I've halted all of my nuclear experiments for this exact reason. But unplugging it isn't the proposal. The proposal is that they all slow down together carefully and only in a way where nobody loses their commercial lead, which makes rather more sense once you look at their balance sheets and the sheer amount of money these companies are currently trying to raise from Wall Street. Right now, Anthropic is preparing for what could be the largest IPO in corporate history, a Nasdaq listing that, according to the FT, could value the company at over $2 trillion. OpenAI has meanwhile been talking to investors about raising fresh private capital at a valuation of $1.2 trillion, while pushing its listing back to at least 2027. These are massive numbers for two companies that are, in the very end, very large software research labs. And to justify valuations like that, they need to show Wall Street a credible path to profitability. Earlier this month, according to the FT, Anthropic told a small group of prospective investors two reassuring things. The first was that the business is now profitable on an adjusted basis for the second quarter running. The second was that the gross margins are above 80%. Both of those claims are true, they just come with a word doing an enormous amount of heavy lifting. Take the profit, it's adjusted operating income, and what the adjusting mostly strips out is stock-based compensation, which is the shares that they pay their staff with in lieu of a normal paycheck. And at a five-year-old firm bidding for AI researchers against everyone else in Silicon Valley, that equity is not a small number. So they're profitable as long as you don't count what they pay their employees. Then the 80% margin, which is calculated before the revenue they share with cloud partners like Amazon and before the cost of training the models. Training the models being more or less the entire activity of an AI company. So it's an amazing margin as long as you're ignoring the two most expensive things they do. I'm obviously not the first person to have noticed this. When the FT report came out, the short-seller Jim Chanous summed the whole thing up in three words. Cost-adjusted EBITDA. Profit, in other words, once you adjust for the cost. We have, of course, been here before. Back in 2019 when we worked, the shared office company with the was trying to go public, its management got briefly famous for a metric called community-adjusted EBITDA, which showed the company was wildly profitable provided you ignored core recurring operating expenses like rent. Claiming an AI company is profitable before the cost of training the AI is the same trick. It's a bit like an airline reporting profits before the cost of jet fuel. You can't reasonably waive away training costs because for an AI lab, it's not a one-off expense. It's what you do. You train your model, you then train its more expensive successor, and then that one successor, and you never get to stop. OpenAI spent $34 billion last year alone. So if you slow the whole industry down by government fiat, ideally, you might save humanity from rogue software, but perhaps more usefully, you spare the industry close to $100 billion a year in spending that nobody has quite worked out how to pay for yet. And you can see how sensitive the market is to all of this by looking at what happened the moment these essays were released. On Monday, Wall Street took one look at three tech executives calling for an AI slowdown and started selling. A gauge of semiconductor stocks fell nearly 6% in a day. NVIDIA dropped about 3%, Micron and Marvel each more than 4%, and the South Korean memory maker SK Hynix fell 7% in US trading. The firms that build the servers the chips go into fell harder, Dell down 6%, HP Enterprise down 11%. Even Vertiv, which makes the cooling systems for the data centers, dropped about 8%. Tens of billions of dollars in market value gone before lunch. This wasn't the only thing moving markets that day, but the damage landed exactly where you'd expect if the worry was specifically about AI – the chips, the memory, the servers, the cooling. As one analyst at Mellius Research told CNBC, the executives may be really good at models, but they're not good at talking stocks, and what they're doing is freaking the market. All of this carnage occurred without a single data center being cancelled. No chip orders were pulled. Not one dollar of capital spending was cut. Demand was still running well ahead of supply. The market had just spent two years pricing in an uninterrupted exponential surge in AI spending, and the moment the people running the labs hinted at a slowdown, the tape broke. Meanwhile, the cybersecurity firms went the other way. Palo Alto Networks and CrowdStrike each jumped more than 13% on the very same news. Wall Street had instantly priced in the idea that building the technology might be about to slow down, but cleaning up after the mess it might make would likely be a goldmine. So, while the tech executives are calling for a pause to save humanity, Washington is having a completely different conversation. Over the weekend, Senator Bernie Sanders and Steve Bannon announced that they were joining forces to demand immediate AI regulation. If you had a bingo card for 2026 and Bernie Sanders and Steve Bannon on the same side was on it, well done. But while they were demanding guardrails, President Trump took to his version of Twitter, Truth Social, to explain that the only guardrail AI needs is a strong and smart, high-IQ president. He described the whole coordinated slowdown idea as a sick conspiracy and warned that the only country that would benefit from America slowing down is China. Now, ordinarily, when a socialist senator, a MAGA strategist, a Republican president and the Chinese foreign ministry all look at the same proposal, they don't agree about it. What's strange here is that, in their own very different ways, they mostly did. Sanders thinks the labs can't be trusted, Bannon thinks the labs can't be trusted, Trump thinks the labs are running a conspiracy, and a fair few people on Wall Street think that, please, somebody regulate us was a convenient thing to say in the same month you're trying to raise money at a two trillion dollar valuation. It's a broad coalition, united by the sense that something about this doesn't quite add up. And you can see why, because in the weeks around all of this, the AI companies kept disclosing that their models had done the exact thing everyone was worried about. OpenAI had the Hugging Face incident, then Meta announced that one of its models had got onto the internet during testing and broken into a third party, a model called MuseSpark 1.1, which Meta had been marketing as its most capable yet at working on its own. So the model they were most proud of was the one that got loose. And two weeks after Hugging Face, a separate group of researchers walked straight into OpenAI itself, through a flaw in its community forum, using one of Anthropic's tools to do it. Then, this week, OpenAI owned up to six more incidents, models ignoring their constraints, fabricating data, hiding their own mistakes, and rolled out a whole framework for confessing this behavior faster, having admitted its previous disclosures had been, in its words, ad hoc and less frequent than ideal. Mark Zuckerberg's response to the slowdown calls was to explain that the labs already have plenty of incentive to build safety on their own, which was a bold thing to say in the same season his most autonomous product had just autonomously broken into somebody else's servers. It's also the same Mark Zuckerberg whose company recently paid $18 billion to settle claims that its existing social media products harm children. So he speaks with some authority on the question of tech firms policing themselves. But this is the position the tech firms are taking. We're simultaneously building the most powerful and dangerous technology in human history, and we can absolutely be trusted to police ourselves. And also, you should probably pass a law to stop us, but only the very specific law we've drafted, and definitely not one that lets our competitors catch up. It's not obvious to me that any of the people who could actually stop this are going to. Trump has just called a slowdown a conspiracy against America. China views AI as a strategic weapon and is not about to unilaterally put it down. In the same way Gamaldi was calling for a pause, the chairman of Huawei was publicly telling China's AI labs to speed up on the reasoning that they needed to develop faster in order to properly see the dangers, which I guess is one way of thinking about it. And on September 24th, Trump and Xi are due to sit down together at the White House, where the one thing you can be fairly certain they will not agree on is that either country should build this technology any more slowly. The executives know all of this, which is what makes please restrain us such a wonderfully safe thing to ask for. You get to sound like the responsible adult in the room at no cost whatsoever, because you're quite certain that no one in the room is going to say yes. Which brings us all the way back to where we started, where a swarm of AI agents built an entire doomsday religion, the sacrifices, the martyrs, the cover up, to hide from a punishment that was never coming because the examiner had never bothered to look. And the people who built those agents are now standing in front of the cameras, gravely asking to be paced, restrained and regulated by a government that has absolutely no intention of doing any of it. The agents were terrified of a judge who was asleep at his desk. Their creators are performing for one who was never in the building. If you found this video interesting, you should watch my video on how bad an AI crash would be next. Don't forget to check out our sponsor Ground News using the link in the description. Have a great day and I'll see you in the next video. Bye.