AWS Details a Nine-Stage Evaluation System for Production AI Agents

AWS published the production case study on October 16

AWS published a detailed evaluation case study on October 16, 2026, for a multi-agent system used by its Sales, Marketing and Global Services Operations team. Field sellers and leaders query customer, revenue and performance data through specialized Performance Insights, Customer Insights and Data Insights agents. Amazon Bedrock AgentCore Evaluations connects the quality loop, with CloudWatch, S3, Redshift, Amazon Quick, Lambda and Bedrock Guardrails supporting telemetry, persistence, dashboards, custom checks and inline filtering.

This is not a new model or API launch. AWS names no foundation-model identifier, context window or inference price for the internal application. The news is the publication of an unusually concrete operating design and claimed production results for an existing AI-agent platform. The service itself is documented as available; the case study does not publish a downloadable implementation or a controlled benchmark.

Green infrastructure hid wrong business answers

AWS says pre-production evaluation found a response that expanded $410.08 into “$410.08 Million,” a metric-label substitution, an annual-recurring-revenue estimate absent from the source data and model calculations presented as sourced facts. Ordinary monitoring remained green because the system returned responses without infrastructure errors. A separate mismatch between coordinated data, tool and prompt changes reportedly cut pass rate from 78% to 38%.

The team converted those failures into six domain-specific categories: metric swap, wrong period, wrong entity, pure hallucination, wrong symbol or unit, and unauthorized calculation. That taxonomy matters more than a generic helpfulness score because each category maps to a business correction and can fail a response independently. The examples and percentages are AWS’s internal account; the post does not provide raw queries, labels or traces for independent review.

Nine stages cover the path from definition to feedback

The framework starts by defining business outcomes, thresholds, latency budgets, cost targets and safety rules. It then compares candidate versions, runs a small merge-time regression set, executes a larger release gate, monitors production canaries, tests adversarial conditions, samples live traffic for drift, exposes trace-level evidence and turns confirmed failures into new regression cases or prompt and tool-description changes.

AWS applies this sequence at three levels: individual tools, each specialist agent and the complete multi-agent orchestration. The evaluator stack mixes deterministic ground-truth checks, built-in relevance, faithfulness and helpfulness judges, custom domain rubrics, and managed third-party evaluators. That separation is useful: exact customer identity, SQL results and financial totals should not be delegated to a probabilistic judge when deterministic comparison is possible.

The release gate is small first, broad second

Every staging deployment runs a 15-case sentinel across quality, safety, latency and cost. AWS says a regression at the tool, agent or multi-agent level blocks the change. Before production, a broader gate replays validated tool responses from a frozen dataset and asks the candidate agent to answer 200 held-out questions end to end. The gate closes if the aggregate score regresses, the candidate fails to beat the production baseline, or too many individual questions miss their threshold.

The frozen replay is a consequential design choice: it lets a team compare agent reasoning without confusing it with a transient tool outage or data change. It cannot replace live integration checks, however. A candidate still needs tests against current authorization, data, tools and failure behavior before release, because a perfect replay can preserve an obsolete contract.

Canaries and sampling trade detection speed for cost and noise

AWS says its batch canary runs every 30 minutes, rotates through the full golden set in 24 hours and pages only after three consecutive windows breach a threshold. The team reports that 15-minute windows produced too many false alarms while 60-minute windows allowed a real problem to persist too long. A full golden-set run executes daily at release-gate thresholds.

For drift, AgentCore Online Evaluation samples 10% of live traffic. AWS describes one customer-identification series falling from 98% to 81% and then 80% across three windows before an alarm. Those settings are workload-specific. A lower-volume or higher-consequence workflow may need deterministic checks on every action; a very high-volume assistant may need stratified sampling so rare tools, regions and user groups are not hidden by the dominant route.

AWS reports gains, not an independently verified benchmark

AWS reports that three months of iteration moved the application’s pass rate from 93% to a stable 96%, reduced pure hallucinations from 138 to under 52 per 10,000 responses, reduced unauthorized calculations from 59 to under 44, and cut mean time to detect from days or weeks to under four hours. It also reports a grounding improvement after replacing real values in few-shot examples with placeholders.

These numbers are valuable operating evidence but remain vendor-reported results from an internal system built on the vendor’s own platform. AWS does not publish the denominator for every result, uncertainty intervals, evaluator-human agreement, raw golden set, model IDs, comparative baseline configuration or complete cost. Several figures explicitly say their displayed numbers are illustrative. Buyers should not read the case study as proof that AgentCore will deliver the same gains on another workload.

Availability is clear; total operating cost is not

The live developer guide documents online, on-demand, batch and dataset evaluation, OpenTelemetry and OpenInference integration, built-in evaluators and private custom evaluators controlled through IAM. It lists a default ceiling of 1,000 evaluation configurations per AWS Region and up to one million input and output tokens per minute per account in large Regions. Teams still need to verify the exact Region, account quota, framework instrumentation and data boundary they intend to use.

AWS’s pricing page lists built-in evaluation at $0.0024 per 1,000 input tokens and $0.012 per 1,000 output tokens. Custom evaluation costs $1.50 per 1,000 evaluations plus model usage; batch evaluation is $0.0018 per 1,000 input tokens and $0.009 per 1,000 output tokens. CloudWatch ingestion, storage, queries and masking are separate, as are runtime, model, Gateway, Guardrails and other application charges. The case study publishes no all-in monthly bill.

What agent teams should do next

Adopt the pattern before adopting the cadence. Name the business failure classes, keep deterministic facts out of LLM-only scoring, freeze a small sentinel for every change, run a wider replay before release and promote confirmed production failures into permanent tests. Version prompts, tools, data snapshots, judge models, rubrics and thresholds together so a score can be traced to the system that produced it.

Trial AgentCore Evaluations when its Regions, telemetry boundary and usage model fit the application. Wait when labels, trace governance or a realistic cost estimate are missing. Reject any plan that treats sampled post-response scoring as authorization or prevention: an evaluator can detect a bad action after the fact, but only application-owned identity, policy, typed validation, spend limits, approvals and idempotency can constrain the action before it takes effect.

Disclosures

AccessAllGPT received no AWS account, AgentCore access, credits, traces, data, private briefing, review, payment or compensation for this article. AWS did not sponsor, review or endorse it. AccessAllGPT did not test, score or rank AgentCore. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with Amazon Web Services. Publication-wide relationships are listed on the disclosures page.