The Truth About Anthropic Researcher is Scary: Why He Quit and What He Knows

Imagine helping build one of the most powerful technologies humanity has ever created, and then one day deciding you can no longer stay silent about what you think it might become. That's essentially what happened this week, when an artificial intelligence researcher named Jacob Coxon walked away from Anthropic, the company behind Claude, and posted a warning that quickly spread across the internet, and what made his message so disturbing wasn't simply For roughly three years, Coxon worked on advanced AI research at two of the most important AI companies in the world. First, OpenAI, the company behind ChatGPT, and later Anthropic, the company behind Claude. These aren't companies watching the AI revolution from the outside, they're helping build it. But after leaving Anthropic, Coxon made his He accused both companies of moving toward increasingly powerful, potentially self-improving artificial intelligence without having solved one of the biggest questions surrounding it. How do you guarantee that something smarter than humans will continue doing what humans actually want? Coxon wrote that neither company, in his view, was acting responsibly. He described the industry as a race toward what researchers call hypothetical AI systems capable of outperforming humans across virtually every intellectual task. And then came the sentence that got everyone's attention. According to Coxon, many of the people building these systems genuinely believe advanced AI could eventually pose an existential threat to humanity. He said these fears aren't simply something discussed by outsiders on the internet. According to him, some researchers and executives working directly on the technology privately worry about the same thing. Now it's important to understand something here. Coxon isn't saying today's ChatGPT or Claude is suddenly going to wake up tomorrow and attack humanity. The concern is about what comes next. Researchers are increasingly discussing something called recursive self-improvement. In simple terms, imagine an AI system becoming capable enough to help improve the next version of itself. That improved version then becomes better at AI research, which helps create an even more capable version, which becomes better at improving itself again. If that cycle happened rapidly enough, intelligence could potentially advance far faster than humans could supervise it. And that's the point at which researchers worry we might lose control. But Coxon's resignation became even more remarkable because another researcher didn't dismiss his claims. Instead, he backed them. Evan Hübinger is Anthropic's alignment science lead. His job focuses specifically on one of the industry's hardest problems, making sure increasingly powerful AI systems remain aligned with human intentions. And after Coxon posted his warning, Hübinger responded publicly. He said Coxon was correct that researchers genuinely worry AI could become catastrophically dangerous. Hübinger went even further. He said his own estimate was that there is a greater than 10% chance that AI could kill all humans within the next decade. Think about that for a moment. Not an anonymous person online. Not someone writing science fiction. One of the people whose job is literally to study how to keep advanced AI under control. And Hübinger admitted something else. He said he believes Anthropic is trying its best. But that the company does not yet have a plan that can guarantee the alignment of a future superintelligence. And according to him, they're not clearly on track to solve it. That doesn't mean catastrophe is inevitable. Far from it. There are researchers who strongly disagree with these predictions. And estimates about existential AI risk vary enormously. Some experts believe scenarios involving human extinction are highly speculative. Others argue that focusing too much on hypothetical future superintelligence distracts from AI problems already happening today. Misinformation, cybercrime, job disruption, surveillance, and misuse. But something unusual is happening. Because Coxon and Hübinger aren't the only people inside the industry calling for caution. Just days before the resignation, OpenAI's chief scientist Jakob Pachacki published an essay about the future of increasingly intelligent machines. He described AI development as moving toward systems that could increasingly participate in their own improvement. And his conclusion wasn't simply excitement about what these machines might achieve. He wrote that this moment calls for extreme caution. Pachacki said he is concerned that no one is fully prepared for the consequences of machine intelligence continuing to rise at its current pace. So we now have researchers connected to multiple leading AI laboratories saying something remarkably similar. AI could become extraordinarily powerful. We don't completely understand what happens if its development accelerates beyond us. And the mechanisms needed to keep future superintelligent systems reliably under human control haven't been fully solved. Which raises an uncomfortable question. If the people closest to this technology are this concerned, why keep building it? Coxon's answer is competition. Every major laboratory knows that if it slows down while another continues, the other company or potentially another country could reach the breakthrough first. And so each participant can believe slowing down would be safer. While also believing that being the only one to slow down would be even more dangerous. It's a technological version of a race where almost nobody feels comfortable stopping. And that's what makes this story bigger than one employee resigning. Artificial intelligence could bring enormous benefits. It may help discover medicines, accelerate scientific research, improve education, and solve problems humans have struggled with for generations. The researchers sounding the alarm aren't necessarily asking humanity to abandon AI. Many of them have devoted their careers to building it. Their warning is that something this powerful shouldn't be treated like an ordinary technology race. Because for the first time in history, humanity may be attempting to create something more intelligent than itself. And whether the darkest predictions ever become reality or not, the fact that some of the people building that future are now publicly asking us to be more careful is probably something worth paying attention to.