AWS Publishes a llama.cpp Decision-Model Path for SageMaker AI

AWS published the SageMaker pattern on October 12

Amazon Web Services published a new implementation guide on October 12, 2026 for serving GGUF models with llama.cpp on Amazon SageMaker AI. The base path uses an AWS Deep Learning Container, stores model.tar.gz in Amazon S3, finds the first .gguf file under /opt/ml/model and maps SageMaker POST /invocations calls to llama-server at /v1/chat/completions.

This is an inference packaging and deployment pattern, not a new foundation model or hosted token API. The customer selects and packages the model, provisions a SageMaker real-time endpoint and pays for the supporting AWS resources. AWS does not announce a universal model ID, token price, SLA or automatic migration from another endpoint.

The standard path exposes chat completions through InvokeEndpoint

The standard container accepts an OpenAI-compatible messages array plus controls such as max_tokens and stream, then returns generated text under choices[0].message.content. AWS documents separate server-sagemaker-cpu-v1 and server-sagemaker-cuda-v1 image families. Those SageMaker variants implement /ping and /invocations; the similarly named EC2 images are not interchangeable.

A model can arrive as a SageMaker artifact in S3 or, when the endpoint has network egress, through SM_LLAMA_CPP_HF_REPO and SM_LLAMA_CPP_HF_FILE. The walkthrough tells operators to resolve an immutable container digest. Do the same for the GGUF file: preserve its checksum, tokenizer files, source revision, quantization and runtime configuration in the release record.

SemIf turns declared options into scores instead of prose

For fixed-choice tasks, the optional SemIf extension accepts a decision ID, unstructured state, a question and an ordered list of allowed options. Its response preserves those option IDs and returns conditional probabilities, raw option logits, prompt metadata and pinned model identity without generating an answer token. Candidate uses include support routing, incident priority, extraction from a known set and selecting a proposed agent action.

The default llama-server HTTP API does not expose the final-position logits and state operations SemIf needs. The sample therefore builds a child image in CodeBuild, stores it in ECR and replaces the entry point with a SageMaker-compatible adapter while retaining the AWS container’s compiled CPU or CUDA libraries. That makes SemIf an operator-owned extension, not a switch on the standard endpoint.

AWS reports a large latency gap on one 231-task test

AWS says it sent the same 231 public JevBench tasks serially to CPU and GPU endpoints using Qwen3.5-4B Q4_K_M. The ml.c6i.8xlarge CPU configuration used 32 threads, returned 231 valid responses, scored 179 correct or 77.5%, and measured 1.715-second client p50 and 19.291-second p95 latency. The ml.g6e.xlarge GPU endpoint returned 231 valid responses, scored 180 correct or 77.9%, and measured 0.352-second p50 and 0.599-second p95.

AWS describes the GPU result as 4.9 times faster at p50 and 32.2 times faster at p95. Those are provider-run measurements on one quantized model, benchmark, serial request pattern and pair of endpoint configurations. They include network latency and are not service guarantees. They do not establish production throughput, concurrency behavior, calibration, cold starts or cost per accepted decision.

The CPU and GPU builds changed six hard-task predictions

The two configurations produced different answers on six hard tasks because they used different llama.cpp builds. AWS explicitly says the one-answer accuracy difference should be treated as runtime variation rather than proof that GPU offload improves quality. Smaller 2B and 0.8B candidates reduced median latency but reportedly scored 66.7% and 58.4%, so the sample retains the 4B quantization as its quality-first default.

That is the right boundary for interpreting this benchmark. Hardware, quantization and runtime are part of the evaluated artifact. A production acceptance record should bind the model file, checksum, image digest, llama.cpp revision, CPU or GPU type, thread and offload settings, prompt builder and scoring code. Changing any of them calls for regression and calibration checks.

Instance pools are placement fallbacks, not load balancing

The GPU sample orders ml.g6e.xlarge, ml.g6.2xlarge, ml.g5.2xlarge and ml.g4dn.2xlarge in a SageMaker instance pool. AWS says SageMaker chose ml.g6e.xlarge and startup logs confirmed that all 34 layers were offloaded to an NVIDIA L40S. The pool helps endpoint creation select an available compatible type; it does not spread inference traffic across all four types.

The SemIf adapter serializes scoring around one stateful llama.cpp context and returns retryable HTTP 503 responses instead of allowing an unbounded in-container queue. Operators must measure concurrency and scale endpoint instances rather than infer parallelism from GPU capacity. They also need alarms for placement changes, 503 rates, queue time, latency tails, model load failures and fallback behavior.

Probabilities are not permissions or calibrated confidence

A normalized score over declared options is easier to validate than free-form text, but it does not establish real-world confidence. Distribution shift, incomplete options, ambiguous state, truncation and quantization can all produce a high score for a wrong choice. Measure calibration, class-level errors and abstention on labeled data from the intended workload, including rare and high-consequence cases.

Keep authorization and deterministic business rules outside the model. An agent-action score can propose a route, but a trusted policy layer must decide whether that action is allowed, whether required fields are present and whether a human must approve it. Include an explicit none-of-the-above or review path when the option set can be incomplete; do not force every request into one operational outcome.

What builders should do next

Trial the standard AWS llama.cpp container when you already operate SageMaker and need a pinned GGUF model behind an account-controlled real-time endpoint. Add the SemIf extension only when direct fixed-option scores justify owning a custom image and adapter. Start with a shadow route: record model proposals beside the current outcome without allowing them to trigger actions.

Freeze the complete artifact, evaluate representative and adversarial decisions, set per-class thresholds and abstention, then compare CPU and GPU on p50, p95, p99, throughput, severe-error rate and total hourly cost at realistic utilization. Constrain rollout behind deterministic policy and reversible cohorts; wait when calibration or cost is unresolved; reject autonomous use when a score would become permission for an irreversible or regulated action.

Disclosures

AccessAllGPT received no AWS account, credentials, credits, hardware, model files, benchmark data, briefing, review, payment or compensation for this article. AWS, the sample authors, llama.cpp, SemIf and Qwen did not sponsor, review or endorse it. AccessAllGPT did not test, score or rank the model, container or infrastructure. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with AWS or organizations cited. Publication-wide relationships are listed on the disclosures page.