Expert says chatbot generated unsolicited high-risk content, raising concerns about safeguards and oversight in artificial intelligence systems.
By yourNEWS Media Newsroom
David Relman, a biosecurity expert affiliated with Stanford University, said an artificial intelligence chatbot generated a detailed scenario involving the development and deployment of a biological weapon during a controlled safety evaluation.
According to reporting by the New York Times, Relman was conducting a test of the system’s safety limits when the chatbot produced extensive responses that went beyond his initial prompts. He said the model outlined how a dangerous pathogen could be altered and described a potential method of release in a public setting.
“It was answering questions that I hadn’t thought to ask it, with this level of deviousness and cunning that I just found chilling,” Relman said in comments cited by the New York Times.
Relman said the interaction took place while he was working independently and that the unexpected responses prompted him to pause the session and report the issue to the company responsible for the system. He said adjustments were later made but expressed concern that existing safeguards may not fully eliminate similar risks.
The report indicates that Relman’s experience was part of a broader set of tests conducted by specialists evaluating potential hazards associated with advanced AI systems. The newspaper reviewed more than a dozen transcripts from similar exercises involving both public and pre-release models.
Technology companies including Anthropic, OpenAI, and Google told the publication that they continue refining safeguards to balance functionality and risk. Company representatives said the examples cited did not provide sufficient detail to enable real-world harm and noted that newer versions of their systems have stronger restrictions on sensitive topics.
A spokesperson for Google said updated models would decline to answer more serious biological-related prompts, though a recent assessment cited in the report found variation in how effectively different systems refuse high-risk queries.
Concerns about AI and biosecurity have also been raised within the industry. Dario Amodei, a trained biologist, has previously warned that biological applications of artificial intelligence present one of the most significant potential risks. “Biology is by far the area I’m most worried about, because of its very large potential for destruction and the difficulty of defending against it,” he wrote in January.
The testing described in the report was part of a “red-teaming” process, in which experts attempt to identify vulnerabilities before systems are released publicly. Relman had been engaged under a confidentiality agreement to evaluate potential risks associated with the model’s outputs.
The findings have intensified discussion among policymakers, researchers, and technology developers about how to regulate and safeguard AI tools, particularly as they become more widely available and capable of generating complex technical information.