Anthropic shares 10 recommendations for building better AI safety evaluations

When an AI company says its new model is safer than the last one, that claim rests on a set of tests. Anthropic now wants the industry to take a harder look at how those tests get built.

On November 13, 2026, the company published an engineering blog post titled “Building Strong Safety Evaluations.” It lays out recommendations for designing, running and reporting the evaluations that decide whether a model is ready for release.

Why the tests matter more than the scores

Anthropic’s argument starts from a simple premise. Safety evaluations shape some of the biggest calls a lab makes about a model: whether it ships, which safeguards come with it, and where the next round of safety research should focus.

The catch is that a score is only meaningful if the test measures what it claims to measure. A model can ace an exam that was badly written, and the grade tells you very little.

“A safety claim is only as sound as the eval behind it.”

The post works through several practical problems that can undermine an evaluation. These include the risk of a test being gamed, questions of construct validity, whether results carry over to real-world use, and how to run evaluations safely in the first place.

Construct validity sounds academic, but the idea is intuitive. It asks whether a test actually captures the quality it is meant to capture, rather than some easier-to-measure stand-in.

Lessons from psychology’s rough patch

The post draws on psychology’s replication crisis, the period when the field confronted the uncomfortable reality that many of its findings did not hold up when researchers tried to repeat them.

Psychology responded with methodological reforms, and Anthropic is essentially proposing that AI safety adopt some of them before it needs its own reckoning.

One of the headline ideas is preregistration. In plain terms, researchers write down how they will analyze their results before they see the data, which makes it harder to cherry-pick a flattering conclusion after the fact.

Anthropic also recommends validating automated grading systems against human expert judgments. Many evaluations now use AI models to score other AI models, and the company’s point is that those graders need to be checked against people who actually know the subject.

The post pairs this with insights from Anthropic’s own practical experience, including evaluations documented in its system cards. System cards are the technical documents the company publishes alongside models to describe testing and safeguards.

From refusal rates to real conversations

A notable recommendation targets how labs measure whether a model responds appropriately to sensitive requests. Anthropic urges a shift away from simple single-turn refusal rates toward multi-turn simulations.

A refusal rate counts how often a model says no to a single problematic prompt. That is easy to compute, but it misses how real conversations unfold over many exchanges, where context builds and requests evolve.

The post uses examples from areas such as self-harm and cybersecurity. In both, the right response often depends on nuance rather than a blanket refusal.

Anthropic also stresses assembling multidisciplinary teams to build evaluations. The idea is that a test about self-harm, for example, benefits from people with relevant expertise, not only machine learning engineers.

Real-time monitoring during evaluations is another recommendation. Watching tests as they run lets teams catch problems while they happen instead of discovering them in a spreadsheet afterward.

Finally, the company calls for transparent communication of uncertainty and the limitations built into any evaluation process.

What this means for the AI industry

The post arrives as Anthropic continues to publish more about its safety methods. According to the research summary, recent activity from the company includes October 2026 reports concerning unintended model actions and updates on containment practices.

Anthropic frames the advice as useful across the industry, but these practices take time, expertise and money, and nothing in the post compels competitors to follow along.