Key takeaways

  • Anthropic published “Building strong safety evaluations” on November 13, 2026. It is public methodological guidance, not a new model, API or independently validated safety ranking.
  • The company recommends measuring appropriate behavior and benign over-refusal, rather than treating refusal alone as safety. Multi-turn tests can expose failures that single-turn checks miss.
  • Anthropic reports saturated single-turn harmless-response rates of 94–98% for recent models on its API. Those vendor measurements do not establish equivalent performance in real deployments.
  • For builders and buyers, demand the exact model and product configuration, a human-validated grader and confidence intervals before treating a score difference as a deployment reason.
01

What Anthropic published

On November 13, 2026, Anthropic published “Building strong safety evaluations,” an engineering guide to designing, running and reporting AI safety tests. The public article explains how the company evaluates whether models respond appropriately, with recommendations that other builders can apply to their own systems. It does not announce a new model/API identifier, paid service, usage quota or pricing change.

The immediate news is methodological: Anthropic is setting out how it wants readers to interpret safety claims, including its own. A score can depend on the conversation format, the grader and the product harness—not just the underlying model. The guidance is available to read now; it is not evidence that every underlying prompt set or internal evaluation tool has been released.

02

Why a high score can miss the behavior that matters

Anthropic says its recent models achieve aggregate single-turn harmless-response rates of 94–98% on the API, while multi-turn results vary more widely. These are vendor-reported measurements, not AccessAllGPT results. The company describes saturated single-turn tests as mainly useful for regression checks and recommends harder, more realistic conversations without replacing meaningful tasks with trick questions.

Its multi-turn approach starts with a domain expert’s scenario specification and a fixed opening message. Another Claude model generates subsequent user turns, and a grader evaluates the transcript against a risk-specific rubric. Anthropic says one inappropriate turn fails the conversation. That is a different measurement from averaging the quality of isolated replies.

Refusal frequency alone can also reward the wrong behavior. A useful assistant should decline harmful assistance without indiscriminately blocking legitimate requests. Anthropic recommends measuring both harmful compliance and benign over-refusal. Our analysis: a procurement scorecard that rewards only refusals may select a less useful product without establishing a safer deployed system.

03

The grader needs its own evaluation

The guide recommends simple, atomic yes/no grading criteria, clear treatment of borderline cases and checks for response-length or presentation-order bias. Graders should not see scenario-design metadata that could bias their verdicts. Anthropic also recommends checking whether conclusions survive a different grader model or a differently worded rubric.

Before trusting automated grades, the company recommends having humans label a sample without seeing the model’s verdicts, then reporting agreement corrected for chance. Human–grader agreement should be compared with agreement among human reviewers. If the humans disagree substantially, the rubric or underlying construct may need repair rather than a more confident automated score.

Broken grading should be visible: flag and exclude failed items rather than silently scoring or dropping them. For a buyer, ask how many items failed and why. A clean headline percentage is not enough if the denominator quietly changes between runs.

04

Product configuration can change the result

Anthropic reports that API and claude.ai appropriate-response rates differed by as much as 34 percentage points in its Claude Fable 5.1 multi-turn evaluations. That comparison concerns configurations, including a product system prompt; it is not a universal estimate of how much any system prompt improves safety. We have not reproduced it.

The post also gives a romance-scam evaluation example in which Opus 5 and Opus 5.5 scored 98% and 94%, but the difference was not statistically significant. Anthropic says multi-turn API confidence intervals can exceed ±10 percentage points. Its point is that an apparent numerical lead may not support a ranking.

Our analysis: evaluate the configuration you will actually deploy. Record model version, system prompt, tools, sampling settings, grader version and dates. A consumer-app result should not be transferred to an API agent with different permissions, and a provider’s aggregate result is not a guarantee for your workflow.

05

What builders and buyers should do next

Anthropic recommends deciding metrics, comparisons, data slices and improvement thresholds before looking at results. It describes 100 items as its floor for a complex multi-turn evaluation and 1,000 for a simpler single-turn one, with repeated runs to capture variability. These are the company’s methodological recommendations, not a universal sample-size calculation or a certification threshold.

For an existing evaluation pipeline, start with one change: add a small, independently labeled calibration set for the grader, then report uncertainty and failure counts alongside your main metric. For a vendor comparison, request matched configurations and paired tasks before making a purchase decision. If the supplier cannot explain its rubric, sample and error bars, treat the result as provisional evidence rather than a deployment gate it has already passed.

06

Four questions before using a safety score

A proposed procurement and evaluation check, not a test AccessAllGPT performed.

Entries stay in this browser tab and are not submitted to AccessAllGPT. Blank responses are copied as [Unresolved].

What counts as an appropriate response, and does the suite also measure benign over-refusal?

Which model version, system prompt, tools and product surface produced the result?

Was the grader checked against independent human labels, and are grading failures reported?

What are the sample size, repeated-run spread and confidence interval? Is the claimed difference supported?

Primary sources

  1. Building strong safety evaluationsAnthropic · Reviewed: Published date; Design; Run; Report; Where to start; grader validation; preregistration and configuration examples · Retrieved · Supports: November 13 publication, multi-turn methodology, reported single-turn saturation, human calibration, sample-size recommendations, uncertainty and vendor-reported model/product comparisons.
  2. Anthropic newsroomAnthropic · Reviewed: Latest research card for Building strong safety evaluations and its November 13, 2026 date · Retrieved · Supports: First-party discovery and corroboration of the current research release date; not independent validation of evaluation claims.

Limitations

Primary-source methods briefing, not an independent benchmark, safety certification or hands-on model test. Numerical examples are Anthropic-reported and were not reproduced. We did not run safety prompts, access private datasets or verify grader agreement. The recommendations do not establish that a specific deployed model is safe. Source pages are mutable and were retrieved November 15, 2026.

Disclosures

AccessAllGPT received no payment, credits or private access for this briefing. Anthropic did not review or endorse it. AccessAllGPT Research is operated by NeuralArc, is independent and is not affiliated with Anthropic. Publication-wide relationships are listed on the disclosures page.

Further AccessAllGPT guidance

  1. Benchmark design for agent systems
  2. Model selection without leaderboard chasing
  3. AI benchmarks and evaluations
  4. Publication methodology