AWS Launches a Ray LLM Container for Multi-Node Inference

AWS launched the multi-node Ray LLM container on October 12

Amazon Web Services published the multi-node Ray LLM Deep Learning Container on October 12, 2026. The container assembles Ray Serve LLM, vLLM, NVIDIA’s NIXL transfer library and AWS’s EFA communication stack so a distributed model-serving application can be declared in YAML and exposed through an OpenAI-compatible endpoint.

The live AWS image catalog lists Ray 2.58, Python 3.13 and CUDA 13.0 for the serve-llm-cuda-v1.0 tag. This is a deployable container image rather than a managed token endpoint: there is no new foundation-model ID, per-token tariff, service-level agreement or AWS-operated autoscaling contract in the announcement.

The reference splits Qwen3.5-9B across two L4 GPU nodes

AWS’s linked sample deploys Qwen/Qwen3.5-9B on Amazon EKS. One g6.8xlarge worker performs prefill, another g6.8xlarge worker performs decode, and a CPU-only m7i.xlarge node hosts system workloads and the Ray head. Each G6 worker supplies one NVIDIA L4 GPU; the manifest sets bfloat16, a 4,096-token maximum model length, 90% GPU-memory utilization and eager execution.

Prefill processes the prompt and creates the key-value cache. Decode consumes that cache while producing tokens. NIXL transfers the cache between workers through LIBFABRIC over EFA. KubeRay 1.4.0 manages the Ray 2.58.0 service, and build_pd_openai_app presents /v1/models and /v1/chat/completions without a custom request handler.

A working image does not remove the distributed-systems boundary

The sample requires an AWS account with billing, AWS CLI credentials, kubectl, eksctl, Helm and envsubst, plus quota for two g6.8xlarge instances—at least 64 vCPUs in the selected Region. GPU workers must remain in one Availability Zone because EFA does not cross AZ boundaries. The scripts also create private subnets, outbound NAT, an EKS control plane, a CPU node and the GPU node group.

The worker container requests one GPU and one EFA interface and runs privileged because the documented NIXL path registers CUDA memory with the network device. That is a meaningful security boundary, not a harmless implementation detail. Teams should threat-model the pod, node and image supply chain; restrict workload identity and network egress; test whether narrower permissions become supported; and avoid mixing unrelated tenants on the same workers.

The transport can silently fall back to TCP-speed behavior

AWS warns that omitting backends: ["LIBFABRIC"] lets NIXL default to UCX at TCP speed even when EFA is present. The sample therefore sets FI_PROVIDER=efa and tells operators to inspect worker logs for the instantiated backend. A healthy HTTP response alone does not prove that requests used the intended interconnect.

Make that verification a deployment gate. Record EFA device availability, the NIXL backend, cross-node bandwidth, KV-transfer time and errors before collecting model latency. Alert on pod rescheduling, backend changes and degraded bandwidth, because a silent transport fallback can erase the reason for separating prefill and decode while preserving apparent endpoint availability.

AWS has not published the comparison needed to justify disaggregation

The post explains why separating compute-heavy prefill from memory-bandwidth-bound decode can reduce interference and permit independent scaling. It does not publish a colocated baseline, request distribution, concurrency, throughput, time to first token, inter-token latency, tail latency, transfer overhead, output quality or utilization result for the reference deployment.

AWS says model-weight loading for Qwen3.5-9B on g6.8xlarge drops from about 145 seconds to about two seconds on restart when the operating-system page cache can retain the roughly 18 GiB checkpoint. That is a provider observation about restart loading under the sample’s memory settings, not a claim that requests become 72 times faster or that the model is production-ready.

The bill is infrastructure-shaped, not token-shaped

AWS’s EKS pricing page lists a standard-support cluster at $0.10 per cluster-hour and separately charges for worker nodes and supporting resources. The sample adds two g6.8xlarge instances, one m7i.xlarge instance, storage, NAT and data processing, ECR access, logging and any transfer or public-address charges that apply. Regional capacity and prices vary.

Neither the launch post nor the sample publishes an all-in hourly, per-token or per-accepted-response figure. Price the exact Region and uptime before trial, including idle GPU time and duplicate model memory. Compare it with a colocated configuration and a managed endpoint at the same quality and latency gates; a preassembled container can reduce integration work without reducing the steady-state infrastructure bill.

What inference teams should do next

Start with one representative model and traffic trace. Pin the sample commit, resolve the mutable image tag to a digest, record the model revision and license, and run a colocated baseline beside the disaggregated configuration. Measure accepted outputs, time to first token, inter-token latency, p50/p95/p99 completion latency, throughput, KV-transfer time, GPU and network utilization, failures, restart behavior and full cost per accepted response.

Trial when long prompts and decode contention create a measured bottleneck. Constrain rollout to one workload and Availability Zone with explicit capacity, spend, privilege and rollback limits. Wait when EFA capacity, image support or the cost comparison is unresolved. Reject the added topology when transfer overhead, operational burden or idle duplication outweighs the measured gain.

Disclosures

AccessAllGPT received no AWS account, credentials, credits, hardware, model files, benchmark data, briefing, review, payment or compensation for this article. AWS, Ray, Anyscale, NVIDIA, vLLM and Qwen did not sponsor, review or endorse it. AccessAllGPT did not test, score or rank the model, image or infrastructure. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with AWS or organizations cited. Publication-wide relationships are listed on the disclosures page.