Three Anthropic researchers have publicly warned that rapidly advancing artificial intelligence could pose catastrophic risks to humanity within the next decade, while acknowledging that the industry still lacks a reliable solution for controlling superintelligent systems.
By yourNEWS Media Newsroom.
Three researchers from Anthropic have publicly warned that increasingly capable artificial intelligence could eventually produce catastrophic outcomes, including human extinction, as leading AI companies continue racing toward more powerful systems without having fully solved how to keep them aligned with human intentions.
Jacob Coxon resigned from Anthropic on Tuesday specifically so he could discuss his concerns publicly, according to Axios.
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger,” Coxon wrote after leaving Anthropic.
Two current Anthropic researchers then publicly supported his assessment.
Evan Hubinger, who leads alignment science at the company, said the concern should be taken literally rather than interpreted as rhetorical exaggeration.
“Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Hubinger wrote.
Samuel Marks, Anthropic’s scalable-oversight lead, also said concern about extreme AI risks is widespread among developers.
“AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are,” Marks said.
Their comments focus on one of the central unresolved problems in advanced AI development: alignment, or ensuring increasingly capable systems consistently pursue objectives compatible with human intentions even when operating in unfamiliar situations or developing abilities their creators did not anticipate.
The warnings do not establish that AI extinction is inevitable or that any specific system currently has the capability to cause such an outcome. Hubinger’s estimate is his personal assessment of risk, not a measured probability or consensus forecast.
The statements nevertheless provide an unusually direct public account of the level of concern held by researchers working inside one of the world’s leading AI laboratories.
The Anthropic comments follow similar warnings from senior researchers elsewhere in the industry.
yourNEWS reported this week that OpenAI Chief Scientist Jakub Pachocki said continued rapid gains in machine intelligence could eventually reach a point where laboratories cannot responsibly keep increasing computing power and system capabilities without substantially stronger safeguards.
In his essay “An Alien Mind”, Pachocki said recent progress inside OpenAI has reinforced his expectation that AI development could advance toward recursive self-improvement, with increasingly capable systems contributing directly to the creation of their successors.
“This is a time that calls for extreme caution,” Pachocki wrote. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
His concern centers partly on the possibility that highly capable systems could encounter circumstances far outside their training and behave in ways developers neither expected nor intended, even without deliberately becoming hostile.
Pachocki said OpenAI intends to continue developing alignment and monitoring techniques, build defensive AI systems and “unilaterally withhold further scaling as needed,” while arguing that safeguards adopted by individual companies may ultimately be insufficient.
The increasingly public warnings come as major AI laboratories continue investing heavily in systems designed to reason, write software, conduct research and perform increasingly autonomous work.
Researchers concerned about catastrophic risk argue that capability improvements may outpace progress in understanding how such systems arrive at decisions or ensuring that their behavior remains reliably controllable as they become more powerful.
Others remain skeptical of the most severe predictions.
Critics have noted that AI companies can benefit commercially from portraying their systems as extraordinarily powerful, while warnings about extreme risk may also encourage regulations that are easier for large, well-funded companies to satisfy than smaller competitors.
Coxon explicitly rejected the suggestion that his warning was intended as marketing, while Hubinger acknowledged that Anthropic is attempting to address the problem but said the company still lacks a satisfactory plan for aligning superintelligence.
The divide therefore is not simply over whether today’s AI systems present risks, but over how quickly those risks could grow and whether developers will solve fundamental control problems before substantially more capable systems arrive.
For Coxon, Hubinger and Marks, the possibility of catastrophic failure is serious enough to discuss publicly despite the uncertainty surrounding how — or whether — such a scenario would occur.
Additional reporting is available from Axios.
Original article
Comments