AWS Publishes a Dual-Monitoring Blueprint for Production AI Agents

What AWS published on September 11

AWS published a production monitoring blueprint for multi-agent systems on September 11, 2026. The reference implementation uses a four-agent airline reservation system built with Strands Agents and hosted on Amazon Bedrock AgentCore Runtime. Its monitoring path combines AgentCore Evaluations, AgentCore Observability, Amazon CloudWatch and AWS DevOps Agent. AWS also linked the sample code in its aws-samples organization.

This is not a model release. The post names no new foundation-model ID, context window, model benchmark or inference price. It is a platform architecture for teams whose agents can return a technically successful response while still choosing the wrong tool, misunderstanding a goal or failing behind a silent infrastructure dependency.

Quality and infrastructure are different failure domains

The useful design choice is the split. AgentCore Evaluations asks whether the agent interaction was helpful, correct or completed the goal. AWS DevOps Agent asks whether IAM, model invocation, tools, networking and other infrastructure supported execution. A green HTTP status or successful tool span does not answer the first question; a low quality score does not identify the second.

AWS demonstrates the distinction with a revoked bedrock:InvokeModel permission. In its example, the supervisor returns a blank output rather than surfacing a clean 403 to the user. AWS DevOps Agent correlates the request, AgentCore runtime, denied Bedrock call and execution role, then recommends restoring narrowly scoped permission. That is a vendor demonstration in AWS’s sample—not measured evidence of diagnosis accuracy or incident-time reduction across other systems.

What AgentCore Evaluations can score now

The AWS post lists 16 built-in evaluators. Thirteen use an LLM judge for dimensions including goal success, correctness, faithfulness, helpfulness, instruction following, refusal, response relevance, harmfulness, stereotyping, conciseness, coherence, tool selection and tool-parameter accuracy. Three deterministic trajectory matchers check whether expected tools appear in any order, exact order or in order with extras allowed.

AWS documents online, on-demand, batch and dataset evaluation surfaces. Online evaluation samples production traces asynchronously; on-demand evaluation targets selected sessions; custom evaluators can use a team’s own rubric. The developer guide says OpenTelemetry and OpenInference instrumentation can normalize supported framework traces. Custom evaluator resources are private and governed through IAM policies.

Sampling controls coverage, latency and spend

AWS says online sampling can be configured from 0.01% to 100% of traces and runs outside the user-facing response path. That avoids adding evaluator latency to the original interaction, but it also means a bad response can reach the user before it is scored—and unsampled failures can remain unseen. AWS’s own post recommends treating asynchronous evaluation as complementary to inline safeguards.

Set sampling from a risk and volume model, not from a round default. Sample high-consequence actions, new releases, rare tools and recently changed routes more aggressively; retain deterministic counters for every transaction; and route confirmed failures into a frozen regression set. A single aggregate score can hide a severe failure concentrated in a small user or tool segment.

The published price is only the evaluator line item

AWS’s pricing page lists built-in evaluator processing at $0.0024 per 1,000 input tokens and $0.012 per 1,000 output tokens, with model usage included. Custom evaluations cost $1.50 per 1,000 evaluations, while the customer also pays for the chosen model. Batch evaluation is listed at $0.0018 per 1,000 input tokens and $0.009 per 1,000 output tokens.

Those rates are not an end-to-end monitoring budget. AWS says traces and evaluation results flow through CloudWatch, where ingestion, storage, queries and sensitive-data masking are billed separately. Agent runtime, model calls, Guardrails and the broader application add their own costs. Forecast using real trace lengths, evaluator count, sampling policy, retention and query volume; re-check the live page because rates can change.

LLM judges are signals, not ground truth

AWS explicitly cautions that LLM-based scoring lacks ground truth and should be calibrated with domain experts. A judge can share blind spots with the system being judged, misread a policy, reward style over task completion or drift when its underlying model changes. Explanations are useful for triage, but they do not turn a probabilistic rating into an authoritative incident label.

Before alerting on a metric, build a human-labeled calibration set with ordinary cases, consequential edge cases and known failures. Measure disagreement by slice, version the rubric and judge configuration, and require a release gate based on the failure classes that matter—not only the average score. Keep deterministic checks for tool identity, parameters, authorization, financial totals and other facts the host can compute.

Do not let post-response monitoring become a safety claim

AgentCore Evaluations observes sampled traces after or alongside execution. It cannot retract an email, reverse a purchase, prevent a data disclosure or guarantee that an unscored interaction was safe. AWS points to Bedrock Guardrails for inline content, topic, grounding and sensitive-information controls, but those controls also do not replace application-owned authorization and transaction validation.

Put non-bypassable policy at the consequence boundary: authenticate the actor, authorize the exact operation and resource, validate typed parameters, cap spend and rate, require informed approval where needed, and make retries idempotent. Monitoring should explain and improve the system around those controls. It should never be the only barrier between a model proposal and an irreversible action.

The sample is a starting point, not production evidence

AWS’s public sample is useful because it exposes a concrete integration path rather than only a diagram. The reference system includes a React frontend, AgentCore Runtime, Identity and Observability, CloudWatch, a Strands swarm and a signed webhook into an AWS DevOps Agent space. AWS notes that webhook credentials are currently generated through the console even though Agent Space management can be automated with CDK and the SDK.

AccessAllGPT recorded the repository route from the AWS article but could not retrieve it directly because outbound GitHub connections failed. Before reuse, pin dependencies and infrastructure modules, inspect IAM grants and webhook verification, decide what trace content may enter CloudWatch, test tenant isolation and deletion, and prove that diagnostic access is read-only where possible. Broad log access given to an incident agent creates its own sensitive-data and authority boundary.

What builders should do next

Pilot the pattern when you already run agents on AgentCore and lack a reliable way to separate task-quality regressions from infrastructure faults. Start with one consequential workflow and three measures tied to outcomes, not all 16 evaluators. Add a known IAM break, a wrong-tool case, a plausible but unsupported answer and a handoff failure; verify which layer detects each case and how long detection takes.

Wait or adapt the pattern when traces cannot leave an existing boundary, CloudWatch costs are unknown, human labels are unavailable or the incident agent would need excessive access. Teams on another runtime can still adopt the central idea: keep behavioral evaluation and infrastructure diagnosis as separate evidence streams, join them with stable trace and release identifiers, and preserve deterministic host controls.

Disclosures

AccessAllGPT did not receive AWS credits, an account, early access, a briefing, sample results, compensation or assistance for this article. AWS did not sponsor, review or endorse it. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with Amazon Web Services, Anthropic, OpenAI or organizations cited. Publication-wide relationships are listed on the disclosures page.