An Anthropic researcher has resigned from the artificial intelligence company, warning that leading AI developers are racing toward increasingly powerful systems despite fears that the technology could pose an unprecedented threat to humanity.
Jacob Coxon, who said he spent the past three years conducting pre-training research at Anthropic and OpenAI, announced his departure in a post on X.
Coxon claimed that neither company was acting responsibly and accused the AI industry of “racing straight to self-improving superintelligence and gambling with our lives.”
Superintelligence refers to a hypothetical stage of AI development in which machines surpass human capabilities across virtually every field.
His concerns were echoed by Anthropic Alignment Science lead Evan Hubinger, who responded that AI could potentially pose an extreme threat to humanity.
Hubinger said he personally believed there was a greater than 10% chance that AI could kill all humans within the next decade. He added that while he believed Anthropic was doing its best to address the risks, the company did not yet have a clear solution to the AI alignment problem for superintelligence.
The AI alignment problem refers to the challenge of ensuring that highly capable AI systems remain consistent with human goals, values and safety requirements.
AI experts call for slower development
Coxon is among a growing number of AI researchers and industry professionals calling for greater caution as companies compete to develop increasingly advanced systems.
In July, more than 1,300 employees from frontier AI companies signed a statement warning that AI capabilities could potentially advance faster than humanity’s ability to understand or control the resulting systems.
The statement called on the US government to support international efforts to develop tools capable of deliberately pacing the development of advanced AI.
Anthropic has also acknowledged concerns over the rapid development of frontier AI. In response to the initiative, the company said its own research into recursive self-improvement highlighted the need for tools that could deliberately pace AI development and give society more time to prepare.
OpenAI similarly said that AI development could eventually reach a point where the world needs to regulate or pace the speed of advancement.
UN warns AI could pose existential risk
Concerns about the potential dangers of advanced AI have also been raised by the United Nations.
UN human rights chief Volker Turk warned that artificial intelligence could pose an existential threat to humanity and called for stronger safeguards around increasingly powerful AI systems.
Speaking in Geneva, Turk said he shared concerns raised by industry insiders about the potential risks of advanced AI.
He called for an “all-out effort” to establish strong safety and security guarantees for artificial intelligence before it becomes too late to address the risks.
The warnings come as AI companies continue to compete in developing more capable models, raising questions about whether technological progress is moving faster than the systems and safeguards needed to control its potential consequences.
