Key takeaways

  • AWS announced on November 13 that Data Capture is available on SageMaker HyperPod clusters with the inference operator, configured through InferenceEndpointConfig or JumpStartModel resources.
  • Three independent tiers record different evidence: endpoint payloads, ALB access metadata, and model-pod payloads. Model-pod capture works without SageMaker endpoint registration.
  • AWS’s improvement example uses synthetic support tickets and Qwen2.5-3B-Instruct. Its reported 100% held-out result is not proof of production accuracy or an isolated fine-tuning effect.
  • Start with a small capture pilot, explicit retention and data-use controls, and an end-to-end check that records actually reach S3. A healthy inference endpoint alone does not prove capture works.
01

What AWS announced and what is available now

On November 13, 2026, AWS announced built-in Data Capture for Amazon SageMaker HyperPod: a way to record live inference inputs and outputs to Amazon S3 and use them for model monitoring, debugging and improvement. AWS says it is available today on HyperPod clusters with the inference operator. This is a model-serving capability, not a new foundation model or a new public model API.

Teams enable it with a dataCapture section in an InferenceEndpointConfig or JumpStartModel Kubernetes resource. The linked documentation includes an add-on configuration update and permissions required for existing clusters. Do not assume an older installation already has every prerequisite, or that creating an endpoint automatically grants access to its capture bucket.

The reviewed announcement does not provide an exhaustive regional availability list, a separate per-token API price or a numerical service-wide capture quota. Its deployment example uses an EKS-orchestrated HyperPod cluster. Confirm support for your cluster and Region before planning a rollout.

02

Three tiers, not three copies of the same log

Tier 1 captures inference input and output at the SageMaker endpoint boundary. It requires endpoint registration and is the documented choice for SageMaker Model Monitor compatibility. Tier 2 records Application Load Balancer access logs: request paths, client IPs and latency metadata rather than full inference bodies. Tier 3 records input and output at the model pod and works without SageMaker endpoint registration.

Each tier can be enabled independently. Turning on both payload tiers can store the same interaction at two points in its path; it is not automatically better evidence. Choose the collection point that answers your question. For traffic and error patterns, metadata may suffice. For reproducing a bad model answer or preparing training examples, payload capture is the relevant option.

The documentation sets the default sampling percentage to 100 for Tier 1 and Tier 3, with both input and output captured by default. Tier 3 buffers 10 records per batch by default, flushes after at most 60 seconds, and captures the entire payload unless a maximum size is configured. Oversized payloads are truncated when a limit is set. These defaults can materially change storage volume and whether a later replay is complete.

03

What AWS’s model-improvement example actually shows

AWS demonstrates capture with a Qwen2.5-3B-Instruct support-ticket router served by vLLM. Synthetic traffic introduces a fictional air-conditioner product after deployment. Athena queries expose the new intent, a SQL labeling policy creates training data, and a LoRA fine-tune runs on a second GPU before a replacement endpoint is deployed alongside the first.

AWS reports improvement from 235 of 280 correct classifications, or 83.9%, to 280 of 280 on a held-out synthetic evaluation set. On 24 hand-written probes, it reports improvement from 16 correct to 20, with remaining mistakes including a refrigerator cooling-fan complaint routed as an air-conditioner ticket. These are vendor-reported results; AccessAllGPT did not execute the workflow.

Two limitations matter more than the headline score. The held-out set comes from the same generator as the training data. And the original router has six allowed categories while the replacement adds ProductAC as a seventh category, alongside changed prompts and constrained-output enums. The comparison is an end-to-end workflow change, not a controlled estimate of the benefit of fine-tuning alone. Before paying for training, test the expanded taxonomy and prompt against the unchanged base model.

The companion README also flags a license issue: it describes the example model as using the non-commercial Qwen Research License and tells production users to choose a model whose license fits their use case. Treat the walkthrough as a reference implementation, not blanket permission to deploy its model commercially.

04

Latency and cost claims need workload context

AWS reports essentially unchanged latency in its small routing benchmark: at concurrency eight, p99 was 249.4 milliseconds direct to vLLM and 249.8 milliseconds through the capture proxy, with throughput of 39.1 requests per second on both paths. AWS says round-to-round variation exceeded the observed difference. This does not establish zero overhead for long prompts, streaming responses, larger payloads or a busy production fleet.

The model-pod implementation adds an NGINX proxy, an OpenTelemetry collector and an S3 uploader. AWS lists combined sidecar requests of 250 millicores of CPU and 640 MiB of memory per pod, plus S3 storage and request costs. Athena analysis adds data-scanning costs, and fine-tuning consumes separate compute. No dollar estimate from this example should be treated as your workload’s price.

The complete demonstration needs two available NVIDIA GPUs because serving retains its GPU while training uses another. The capture feature itself should not be confused with that two-GPU training example. Our analysis: the useful buying question is whether better failure visibility justifies storage, collection and review costs—not whether a synthetic router can reach a perfect score.

05

What to check before collecting real interactions

Captured prompts and answers can contain personal information and confidential business data. Establish permission to retain and reuse them, restrict bucket access, set lifecycle rules, and decide what must be redacted before labeling or training. Customer-managed KMS keys are supported for the payload tiers. The documentation says ALB logs use S3 default encryption rather than KMS, and warns that URLs and query parameters can expose sensitive input.

Verify delivery, not just endpoint health. AWS warns that missing permissions for the Mountpoint-S3 CSI driver can leave inference working while no capture files appear. In its specific demo, calls to port 8081 pass through the capture proxy while calls directly to vLLM on port 8000 bypass it. These are walkthrough ports, not a promise about every deployment.

The practical next step is a low-volume pilot with non-sensitive requests. Confirm that the intended path produces readable records after the flush interval, measure capture overhead on representative traffic, and compare a prompt-only fix with fine-tuning on independently reviewed examples. Promote a replacement only after checking both new intents and existing categories, with the previous endpoint available for rollback.

06

A first capture pilot

A proposed acceptance checklist, not a test AccessAllGPT has performed.

Entries stay in this browser tab and are not submitted to AccessAllGPT. Blank responses are copied as [Unresolved].

Choose endpoint payloads for Model Monitor, ALB metadata for traffic analysis, or pod payloads for diagnosis and model improvement. Avoid duplicate collection without a reason.

Approve retention and training reuse, choose sampling and payload limits, restrict access and configure encryption and lifecycle rules.

Send known non-sensitive inputs through the intended route and inspect the corresponding S3 records. Endpoint success alone is insufficient.

Compare taxonomy/prompt changes without training against a fine-tune, use a separate evaluation set, check old categories and keep a rollback endpoint.

Primary sources

  1. Build a model improvement flywheel with Amazon SageMaker HyperPod Data CaptureAWS · Reviewed: November 13 publication date; Three-tier architecture; prerequisites; capture path; Measuring the lift; latency; costs; conclusion · Retrieved · Supports: Availability statement, inference-operator configuration, synthetic support-router example, vendor-reported evaluation and latency results, GPU prerequisites, sidecar resources and capture-delivery caveats.
  2. Data capture for inference on HyperPodAWS Documentation · Reviewed: Tier comparison; Configuring data capture; Tier 1 and Tier 3 defaults; Update Addon; permissions; Best practices · Retrieved · Supports: Independent capture tiers, endpoint-registration distinction, sampling and buffering defaults, truncation behavior, encryption differences, configuration and IAM prerequisites.
  3. Continuous model improvement flywheel with SageMaker HyperPod Data CaptureAWS Labs · Reviewed: README overview; synthetic-data and model-license notice; prerequisites; walkthrough and endpoint evaluation instructions · Retrieved · Supports: Public companion code, synthetic-data disclosure, model-license warning, two-GPU demonstration requirements and distinct category sets for original and replacement endpoints.

Limitations

Primary-source briefing, not hands-on cloud testing, independent performance verification or a license opinion. We did not provision a cluster, capture customer interactions or run the fine-tune. AWS’s metrics concern one synthetic support-routing experiment and do not isolate training from taxonomy and prompt changes. Documentation and the companion repository are mutable; regional availability and a workload-specific dollar estimate were not established.

Disclosures

AccessAllGPT received no payment, credits or private access for this briefing. AWS did not review or endorse it. AccessAllGPT Research is operated by NeuralArc, is independent and is not affiliated with AWS. Publication-wide relationships are listed on the disclosures page.

Further AccessAllGPT guidance

  1. Managed LLM APIs versus self-hosting
  2. Benchmark design for agent systems
  3. AI models and inference coverage
  4. Publication methodology