Nexo Earn with Nexo
Anthropic urges top AI labs to slow development over self-improvement risks

Anthropic urges top AI labs to slow development over self-improvement risks

The Claude maker says the ability to slow global AI development would 'likely be a good thing,' citing internal data on how fast its own models are improving.

Anthropic said the world should have the option to slow or temporarily pause frontier AI development if leading AI labs and governments can create a credible way to verify that everyone is complying.

In a new report from the Anthropic Institute, the company said a pause could help give policymakers, researchers, and civil society more time to address the risks tied to increasingly capable AI systems. But the firm warned that a slowdown would only improve safety if it applied across multiple well resourced labs at or near the frontier.

Anthropic said a unilateral pause by one company would be easier to implement but far less effective, because it could simply hand the lead to less cautious actors. The company said any meaningful pause would require developers in multiple countries to stop under the same conditions and verify that competitors had also stopped.

The warning comes as Anthropic says AI systems are already accelerating the development of new AI models. As of May 2026, more than 80% of code merged into Anthropic’s codebase was authored by Claude, up from the low single digits before Claude Code launched in research preview in February 2025.

Advertisement

The company said the typical Anthropic engineer merged eight times as much code per day in the second quarter of 2026 as they did in 2024. A March 2026 internal poll also found that research staff using Mythos Preview estimated they were producing roughly four times as much output as they would without AI models.

Anthropic said those gains point to a broader shift in how frontier AI systems are built. Human engineers and researchers are still setting goals, reviewing outputs, and deciding which problems matter, but Claude is taking on more of the execution work across coding, testing, debugging, and experiment optimization.

The company framed that trend as an early step toward recursive self improvement, a scenario in which AI systems become capable of autonomously designing and developing their own successors. Anthropic said such a system does not exist yet and may not be inevitable, but warned it could arrive sooner than most institutions are prepared for.

If that happens, Anthropic said the pace of AI development could become tied more directly to compute availability and efficiency gains, while humans shift toward oversight, validation, and verification.

That would make the systems used to monitor frontier AI development more important, especially if AI models begin playing a larger role in building future versions of themselves.

Anthropic said the challenge is that verifying a pause in AI development is harder than monitoring many other technologies. Training runs can be concealed, their inputs are general purpose, and the incentive to quietly defect would be enormous if one actor could gain the lead while others stop.

The company said a credible pause would need clear rules around what triggers it, what ends it, and who decides whether the conditions have been met. It compared the challenge to arms control regimes for other complex technologies, while noting that those systems took decades to build and that AI may not leave governments that much time.

Anthropic said it plans to organize conversations in the coming months with policymakers, researchers, civil society, and other AI companies to examine recursive self improvement and possible coordination mechanisms for frontier AI development.

The company said it would expect to slow or temporarily pause development if other frontier developers also did so in a verifiable way.

Disclosure: This article was edited by Estefano Gomez. For more information on how we create and review content, see our Editorial Policy.

Anthropic urges top AI labs to slow development over self-improvement risks

Anthropic urges top AI labs to slow development over self-improvement risks

The Claude maker says the ability to slow global AI development would 'likely be a good thing,' citing internal data on how fast its own models are improving.

Anthropic said the world should have the option to slow or temporarily pause frontier AI development if leading AI labs and governments can create a credible way to verify that everyone is complying.

In a new report from the Anthropic Institute, the company said a pause could help give policymakers, researchers, and civil society more time to address the risks tied to increasingly capable AI systems. But the firm warned that a slowdown would only improve safety if it applied across multiple well resourced labs at or near the frontier.

Anthropic said a unilateral pause by one company would be easier to implement but far less effective, because it could simply hand the lead to less cautious actors. The company said any meaningful pause would require developers in multiple countries to stop under the same conditions and verify that competitors had also stopped.

The warning comes as Anthropic says AI systems are already accelerating the development of new AI models. As of May 2026, more than 80% of code merged into Anthropic’s codebase was authored by Claude, up from the low single digits before Claude Code launched in research preview in February 2025.

Advertisement

The company said the typical Anthropic engineer merged eight times as much code per day in the second quarter of 2026 as they did in 2024. A March 2026 internal poll also found that research staff using Mythos Preview estimated they were producing roughly four times as much output as they would without AI models.

Anthropic said those gains point to a broader shift in how frontier AI systems are built. Human engineers and researchers are still setting goals, reviewing outputs, and deciding which problems matter, but Claude is taking on more of the execution work across coding, testing, debugging, and experiment optimization.

The company framed that trend as an early step toward recursive self improvement, a scenario in which AI systems become capable of autonomously designing and developing their own successors. Anthropic said such a system does not exist yet and may not be inevitable, but warned it could arrive sooner than most institutions are prepared for.

If that happens, Anthropic said the pace of AI development could become tied more directly to compute availability and efficiency gains, while humans shift toward oversight, validation, and verification.

That would make the systems used to monitor frontier AI development more important, especially if AI models begin playing a larger role in building future versions of themselves.

Anthropic said the challenge is that verifying a pause in AI development is harder than monitoring many other technologies. Training runs can be concealed, their inputs are general purpose, and the incentive to quietly defect would be enormous if one actor could gain the lead while others stop.

The company said a credible pause would need clear rules around what triggers it, what ends it, and who decides whether the conditions have been met. It compared the challenge to arms control regimes for other complex technologies, while noting that those systems took decades to build and that AI may not leave governments that much time.

Anthropic said it plans to organize conversations in the coming months with policymakers, researchers, civil society, and other AI companies to examine recursive self improvement and possible coordination mechanisms for frontier AI development.

The company said it would expect to slow or temporarily pause development if other frontier developers also did so in a verifiable way.

Disclosure: This article was edited by Estefano Gomez. For more information on how we create and review content, see our Editorial Policy.