SGLang-Neuron opens a public preview for serving AI models on AWS Trainium

What became available on November 12

AWS announced on November 12, 2026 that SGLang-Neuron is publicly available through an AWS Deep Learning Container released by RadixArk in collaboration with Annapurna Labs. The plugin lets teams serve AI models on AWS Trainium using SGLang’s familiar launcher and generation APIs. The project README describes the release as a public preview; public access should not be confused with general availability or a production-readiness guarantee.

The published image is public.ecr.aws/neuron/pytorch-inference-sglang-neuron:v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04. AWS’s walkthrough launches openai/gpt-oss-120b and sends a request to /v1/chat/completions. This is software for a customer-operated inference server, not a new foundation model or an AWS-managed model endpoint with a newly announced per-token price.

The practical change is a new hardware path for teams already using SGLang. They can investigate Trainium without replacing the entire client-facing serving workflow. Whether that reduces cost or improves throughput depends on the supported model, instance allocation and workload; this announcement does not settle those questions.

The interface stays familiar; execution moves to Neuron

SGLANG_PLATFORM=neuron selects the out-of-tree platform plugin. AWS says SGLang retains its API server, request lifecycle, scheduler and prefix-cache bookkeeping, while the plugin supplies Trainium-specific execution, attention and mixture-of-experts components. TorchNeuron exposes Trainium as a native PyTorch device; torch.compile prepares PyTorch operations and Neuron Kernel Interface kernels for execution.

Prefill and decode can also be separated. The prefill server computes the prompt’s key-value cache, transfers it through NIXL over Elastic Fabric Adapter, and the decode server generates tokens. That is a documented deployment capability, not evidence that splitting these stages will improve every workload. It brings additional placement, network and capacity choices.

The repository pins upstream SGLang to commit 43881a7 and says a different revision stops launch until the integration is revalidated. Familiar commands therefore do not imply compatibility with any SGLang version. Use the validated environment rather than treating this as an unrestricted plug-in for an existing installation.

Supported models are a catalog, not a blanket promise

The reviewed README says LLMs run on Trn3 instances and diffusion models on trn2.48xlarge. Its LLM table includes openai/gpt-oss-120b, Qwen/Qwen3-8B, Qwen/Qwen3-30B-A3B, deepseek-ai/DeepSeek-V2-Lite, amd/GLM-5.2-MXFP4 and moonshotai/Kimi-K3, among other listed configurations. Architecture support does not establish support for every related checkpoint or every feature of a multimodal model.

Some configurations are substantial deployments: the project describes Kimi-K3 disaggregated serving across three Trainium3 nodes. Its Qwen3.5-397B-A17B entry is explicitly a text-only view of the FP8 checkpoint. Neither should be reduced to “drop in any model.” Check the exact recipe, parallelism, dtype and hardware before planning a trial.

The published requirements include Neuron SDK 2.33.0, a compatible Neuron driver and Docker for the container path; disaggregated serving needs EFA between nodes. The plugin is Apache-2.0, but that license does not replace the terms of the model weights you choose. We did not inspect each checkpoint’s license in this briefing.

Two migration details can break an otherwise familiar client

First, the quickstart defaults to greedy on-device sampling. A request must set temperature to 0 or top_k to 1; the documentation says other requests, including one that omits temperature, are refused. Sampling alternatives require launching with SGLANG_NEURON_SAMPLER=temperature or topk_topp. Test the actual request payload, not only the endpoint name.

Second, compilation happens before the server answers /health. The project says it compiles and warms the serving graphs at startup, with later launches able to reuse the compile cache. Do not mistake a cold compile for request latency or assume a model is ready because the container is running. Persist the documented compile caches and measure cold and warm starts separately.

The README also says the runtime refuses LoRA, embedding servers and SGLang’s weight-update endpoints. That last exclusion matters because AWS’s announcement mentions reinforcement-learning rollouts: generating trajectories and continuously updating serving weights are separate capabilities. The launch wording does not establish an online RL loop compatible with existing weight-refresh code.

What an inference team should do next

Start with one documented model recipe and a non-sensitive evaluation set. Keep the checkpoint, input lengths, output lengths and concurrency comparable with the existing deployment. Check response correctness, sampling behavior, cache reuse and failure handling before comparing time to first token, sustained throughput and total instance cost.

AWS describes Trainium as offering strong economics, but this briefing establishes no numerical price, throughput advantage or saving. Include idle allocation, compile time, the required node count and any disaggregated networking in a cost comparison. Teams that need unsupported adapters, embeddings or weight updates should wait for explicit feature support or keep those workloads on their current serving stack.

Disclosures

AccessAllGPT received no payment, credits or private access for this briefing. AWS, RadixArk and the SGLang project did not review or endorse it. AccessAllGPT Research is operated by NeuralArc, is independent and is not affiliated with these organizations. Publication-wide relationships are listed on the disclosures page.