Key takeaways
- AWS announced managed GPU partitioning for SageMaker AI real-time inference on November 6, 2026. It uses NVIDIA Multi-Instance GPU (MIG) to give separate inference components fixed compute slices and dedicated GPU memory.
- The launch supports P4d/A100, P5/H100, G7/RTX PRO 4500 and G7e/RTX PRO 6000 instance families in seven AWS Regions. It does not cover single-model, serverless, asynchronous or heterogeneous-pool endpoints.
- Builders declare AcceleratorPartitionLayouts on the production variant and AcceleratorPartitionType on each inference component. Invocation remains unchanged, but a running endpoint cannot be re-carved: changing the layout requires a new endpoint and traffic migration.
- MIG isolates GPU compute and memory, not the entire host. Components still share CPU, system memory, storage, networking and endpoint routing, and AWS documentation says partitioned inference components do not publish GPU, CPU or memory utilization metrics.
- AWS bills the whole instance; partitioning does not reduce its hourly rate. Savings depend on filling slices and workload shape. AWS reports one H100 slice beat a full A100 in its Llama 3.1 8B test at sufficiently high load, while the A100 stayed cheaper below the crossover.
- Pilot with a stable portfolio of small and mid-sized models, size slices for weights plus KV cache and runtime overhead, and benchmark latency, throughput and failure rates at production-shaped prompt lengths before migrating traffic.
SageMaker AI launched managed GPU partitioning on November 6
AWS announced on November 6, 2026 that SageMaker AI real-time inference can partition supported NVIDIA GPUs with Multi-Instance GPU, or MIG. A physical accelerator becomes several hardware-isolated slices, and each SageMaker inference component can keep its own model loaded on one slice. The feature is available now through existing SageMaker endpoint and inference-component APIs.
The practical change is operational. Teams previously had to assign a whole accelerator to each resident model, accept the loading delays and shared-device contention of multi-model endpoints, or operate MIG through Kubernetes themselves. SageMaker now handles the carve, placement and scaling on a managed endpoint. This is an inference-platform release, not a new model or a new invocation API.
Two optional fields define the endpoint and model slice
AcceleratorPartitionLayouts is added to a production variant in CreateEndpointConfig. It declares one layout that SageMaker applies uniformly to every GPU on every instance in that variant. AcceleratorPartitionType is added to an inference component’s compute requirements and requests a profile such as mig-4g.20gb. Existing InvokeEndpoint calls continue addressing the inference component by name; the partition is not exposed in the request.
AWS says the feature creates no new API operation, resource type or IAM action. That lowers integration work, but it does not make capacity planning automatic. A requested profile must exist in the endpoint layout, and SageMaker rejects an incompatible request rather than falling back to a whole GPU.
Four GPU families and seven Regions are supported at launch
AWS documents support for P4d instances with 40 GB A100 GPUs, P5 with 80 GB H100 GPUs, G7 with 32 GB RTX PRO 4500 Blackwell GPUs and G7e with 96 GB RTX PRO 6000 Blackwell Server Edition GPUs. Depending on the accelerator, a GPU can be divided into two, four or as many as seven partitions. Profile names and valid combinations differ by family.
The launch Regions are US East (N. Virginia and Ohio), US West (Oregon), Asia Pacific (Singapore and Tokyo), and Europe (Frankfurt and Ireland). Instance availability and account quota remain separate constraints. AWS also warns that the mig-1g.10gb profile on P4d requires a newer inference AMI than the default image; without it, carving can fail without naming the offending profile.
The boundary is GPU isolation, not full tenant isolation
MIG assigns each partition dedicated GPU memory and a fixed share of compute. AWS says one component cannot read another partition’s GPU memory or consume its GPU throughput, which is materially stronger than time-sharing several models on one unpartitioned device. Keeping every model resident also avoids the load-and-evict cold starts associated with a multi-model endpoint.
The host remains shared. CPU, system memory, storage, networking and endpoint routing are outside the MIG boundary, so a slice is not a complete security or tenancy boundary. Teams still need workload isolation, access-control, noisy-neighbor and failure tests above the GPU layer.
Layouts are fixed, and observability has a notable gap
SageMaker chooses the partition layout while provisioning the endpoint. AWS says a running endpoint cannot be re-carved: an UpdateEndpoint request with a different layout returns a 400 error. Moving to another carve therefore means creating a new endpoint, recreating inference components and migrating traffic. A component’s own partition type is also fixed for its lifetime.
AWS documentation says partitioned endpoints still publish invocation and latency metrics, but their inference components do not publish GPU, CPU or memory utilization metrics. That makes application-level throughput, queueing, latency and error telemetry especially important. It also weakens direct evidence about whether an expensive slice is saturated, so teams should preserve load-test measurements and use container-level observability where supported.
AWS reports a workload-dependent H100-slice advantage over an A100
AWS compared one mig-3g.40gb slice of an H100 with one full 40 GB A100, serving Llama 3.1 8B Instruct through identical containers on SageMaker real-time endpoints. Using genai-bench, it swept five input/output shapes and concurrency from 2 to 512. AWS reports that the H100 slice delivered 53% to 89% more output tokens per second at the best-cost point for each scenario, with lower time to first token and inter-token latency. These are vendor-run results, not independent benchmarks.
AWS also reports that the slice cost 18% to 34% less per million output tokens only above workload-specific crossovers: concurrency 64 for 100-token prompts and concurrency 4 for 8,000-token prompts. Below the crossover, the full A100 was cheaper. The comparison derives a fraction of whole-instance on-demand prices rather than changing the AWS bill; customers are still charged for the complete instance.
Slice sizing must include KV cache and runtime memory
A partition has to hold more than model weights. AWS’s example estimates about 16 GB for an 8B model at FP16, then adds KV cache for the configured context and concurrency plus roughly 1–2 GB of serving overhead. A profile sized only for weights may load successfully but fail under traffic or force a shorter usable context.
Compute and memory also bind different phases. Prompt prefill is compute-heavy, while token decode repeatedly reads weights and is often memory-bandwidth-bound. Test the exact mix of input length, output length, batch size and concurrency; a slice that looks efficient for classification may be the wrong shape for long-context generation.
What inference teams should do next
Start with a stable portfolio: embedding models, rerankers, classifiers, guardrails, fine-tuned tenant variants or pipeline stages that leave most of a full GPU idle. Inventory model weights, peak KV cache, runtime overhead, request shape, latency target and region quota. Then choose a valid layout and keep spare capacity for replicas or canaries rather than filling every slice on paper.
Run the same frozen load against whole-GPU and partitioned candidates, including warm-up, sustained traffic, bursts, failures and traffic migration. Measure accepted requests, p50/p95/p99 latency, time to first token, output throughput, deadline failures and whole-instance cost. Adopt MIG only when the complete instance is well packed and the fixed-layout migration burden is acceptable; keep whole accelerators for models that need multiple GPUs or frequently changing capacity.
Copy-ready SageMaker GPU partition review
Use this before fixing an endpoint layout, because changing the carve later requires a replacement endpoint and traffic migration.
Entries stay in this browser tab and are not submitted to AccessAllGPT. Blank responses are copied as [Unresolved].
Models, versions, weights, traffic owners, request shapes, context and output limits, concurrency, latency SLOs and growth window.
AWS Region, instance family, GPU, account quota, inference-component endpoint and unsupported modes that must remain elsewhere.
Weights, KV cache at peak context and concurrency, serving runtime overhead, safety margin and observed out-of-memory behavior.
AcceleratorPartitionLayouts value, every component’s AcceleratorPartitionType, replica count, unused slices and profile compatibility.
GPU guarantees plus controls for shared CPU, host memory, storage, network, routing, credentials, data and tenant boundaries.
Whole-GPU and partitioned variants, identical images and model builds, prompt/output distributions, concurrency ladder and run duration.
Accepted throughput, p50/p95/p99 latency, first-token latency, deadline and OOM failures, whole-instance cost and utilization evidence.
Metrics, logs, alerts, component replacement, endpoint rebuild, blue/green traffic migration, rollback time and capacity reservation.
Adopt, bounded pilot, retain whole GPU or reject; eligible models, owner, expiry and re-test trigger.
Primary sources
Browse the publication-wide evidence index →
- SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpointsAWS Machine Learning Blog · Reviewed: November 6 announcement; availability; supported endpoints, instances and Regions; API fields; partition profiles; billing; limitations; AWS performance and cost comparison; getting started · Retrieved · Supports: AWS announced managed NVIDIA MIG partitioning for SageMaker real-time inference-component endpoints, documented the supported hardware and Regions, and reported a vendor-run H100-partition versus A100 comparison.
- Using GPU partitions on SageMaker AI real-time endpointsAmazon SageMaker AI Developer Guide · Reviewed: Benefits; isolation boundary; API fields; endpoint lifecycle; monitoring; supported instance families, Regions and partition types; P4d driver warning · Retrieved · Supports: AWS documentation confirms the production interface and operating constraints: two optional fields, immutable layouts on running endpoints, no per-component resource-utilization metrics, four supported instance families and seven launch Regions.
- SageMaker GPU partitioning sampleAWS Samples on GitHub · Reviewed: Repository path linked from the AWS launch post · Retrieved · Supports: AWS links this public sample as the starting implementation for deploying several models onto GPU partitions; AccessAllGPT did not execute the notebook.
Limitations
This is a source-backed launch briefing, not an independent SageMaker or NVIDIA MIG test. AccessAllGPT did not have an AWS account, endpoint quota or production workload in this job and did not provision P4d, P5, G7 or G7e capacity; execute the AWS sample; validate profile combinations; inspect hardware isolation; reproduce AWS’s Llama 3.1 8B test; measure latency, throughput, failures, carving time or cold starts; compare Kubernetes, multi-model endpoints or other clouds; test AMI and driver behavior; audit host-level tenancy; or inspect billing. AWS’s performance, cost and operational figures are vendor-reported and workload-specific. Regions, quotas, instance availability, prices, supported profiles and documentation can change.
Disclosures
AccessAllGPT received no AWS or NVIDIA account, credits, hardware, quota, private benchmark data, briefing, review, payment or compensation for this article. AWS, NVIDIA, Salesforce and the sample repository contributors did not sponsor, review or endorse it. AccessAllGPT did not test, score or rank the service. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with AWS or NVIDIA. Publication-wide relationships are listed on the disclosures page.
