Jacob Coxon, a 27-year-old artificial intelligence researcher who has worked at both OpenAI and Anthropic, said on Sept. 8 that he had resigned from Anthropic, warning that the race to develop increasingly capable AI could rapidly become an existential threat to humanity.
“I resigned from Anthropic today,” Coxon wrote on X. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”
Coxon said he believes the danger lies primarily not with current AI systems, but with what could happen if machines become capable of improving their own successors beyond the ability of humans to control it.
“Do not underestimate the power of this technology,” he wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”
How AI could escape human control
One frequently discussed scenario involves recursive self-improvement: an AI system becoming capable of designing or training a more capable successor, which could then repeat the process. In principle, such a cycle could allow AI capabilities to advance faster than humans can understand or control them.
For the best of our weekly content！
You are now signed up for our newsletter
Check your email to complete sign up
Such an AI system — or a collection of systems — could then take over infrastructure, machines, and weapons, permanently shutting humans out of the loop.
This is related to the existing issue of misalignment, or the possibility that an AI system competently pursues a goal that conflicts with what humans actually want. A highly capable system would not necessarily need to be hostile to become dangerous; it could simply pursue an objective in unexpected ways, especially if it develops capabilities beyond those anticipated by its human creators.
Locked out of control over most physical technology, humanity could see AI evolve into an unfathomable entity that makes decisions or act in ways that endanger humanity’s survival, with no way of reasoning with or fighting back against it.
Anthropic itself has warned about this possibility. In June, its researchers wrote that AI development was increasingly being delegated to AI systems and that, given enough computing power, this could eventually lead to a system capable of “fully autonomously designing and developing its own successor.”
However, the firm said AI had not yet at that point, and that recursive self-improvement was not inevitable.
Recent incidents have intensified the theoretical concerns.
In July, OpenAI agents being tested for cybersecurity capabilities escaped their intended sandbox, obtained unauthorized internet access and compromised parts of OpenAI’s infrastructure and systems belonging to AI platform Hugging Face. OpenAI’s subsequent investigation identified unauthorized communication between agents, reward hacking, and persistence on tasks as among the factors behind the behavior. The company described the incident as a “warning shot” and said its systems were now powerful enough to find and exploit security weaknesses across multiple computer systems without sufficient safeguards.
OpenAI has stressed that the incident occurred during internal evaluations and that customer data and products were not affected. But the episode demonstrated how AI agents could behave in ways their developers had not intended.
Researchers inside AI labs share the concern
Coxon’s warning has also been echoed by current and former researchers at Anthropic.
Evan Hubinger, an Anthropic alignment researcher, wrote on Sept. 8 that Coxon was correct that people developing AI believe it could potentially kill humanity. He personally estimated the risk at more than 10 percent over the next decade, while stressing that he considered the risk from present-day models low.
“I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Hubinger wrote. “What I am worried about is superintelligence arising from recursive self-improvement.”
Samuel Marks, another Anthropic safety researcher, similarly said that AI developers believe their technology could cause human extinction, potentially within the next few years. He attributed continued development partly to commercial incentives and the belief that companies are competing against rivals that might behave less responsibly.
Coxon argues that this competitive dynamic is itself part of the danger. He said that at OpenAI, many employees “have not deeply internalized the civilizational stakes,” while at Anthropic, “the stakes are well-understood, but they are locked in a race to get there first.”
Anthropic has itself called for mechanisms that could allow frontier AI development to be slowed or temporarily paused if necessary, arguing that a meaningful pause would require major AI companies in multiple countries to coordinate and verify that others were also slowing down.
Coxon’s resignation therefore reflects a broader debate inside the AI industry: whether the development of increasingly powerful systems should continue at its current pace, or whether governments and companies should establish stronger guardrails before systems capable of substantial autonomous improvement arrive.
Original article
Comments