Key takeaways
- AWS published its distributed KV caching skill walkthrough on November 20, 2026 Pacific time, November 21 UTC; the skill files are publicly accessible in Agent Toolkit for AWS.
- The workflow targets self-hosted vLLM or SGLang fleets with repeated long prefixes, using LMCache and an ElastiCache shared tier rather than recomputing every returning context.
- Production RDMA requires compatible EFA hosts and same-Availability-Zone placement. This is not a general drop-in cache for every inference platform.
- Measure reuse and net costs before creating a cluster. Freed GPU time becomes a financial saving only when it avoids capacity or supports additional useful traffic.
What AWS made available
AWS has published a distributed KV caching agent skill for assessing and configuring shared attention-state caching on self-hosted LLM fleets. Its walkthrough is dated November 20, 2026 in U.S. Pacific time, with a publication timestamp falling on November 21 UTC. The skill is available in the public Agent Toolkit for AWS repository and is intended for coding agents that support Agent Skills, including Kiro, Claude Code and OpenAI Codex.
The new item is an agent-guided assessment and setup workflow, not a new foundation model, a new model API identifier or evidence that distributed caching itself first launched on that date. AWS walks through diagnosis, savings estimation, cluster creation and troubleshooting. The public skill and documentation are directly accessible; we have not checked account-level resource availability or deployed the stack.
Why a shared KV tier matters for long-context inference
During prefill, an autoregressive model computes attention key-value tensors for prompt tokens. A returning chat or agent request may contain a long prefix already processed elsewhere in the fleet. If routing changes or local memory has evicted that state, another GPU may recompute it. A shared tier allows compatible servers to load the stored state instead.
AWS’s architecture puts an LMCache server beside each GPU server and uses ElastiCache for Valkey as the shared L2 tier. Its production path moves data over EFA RDMA. This caches model state, not final answers: it is distinct from response caching, semantic caching and Amazon Bedrock managed prompt caching. The skill explicitly excludes short or never-repeated prompts and output-dominated workloads from its intended fit.
Our analysis: the relevant audience is an inference operator already paying for repeated prefill across replicas. For a team with short requests or effective local prefix caching and session affinity, adding another tier may create cost and operational work without a useful payoff. AWS itself says short-prefix transfers can cost more time than recomputation.
The prerequisites narrow the deployment decision
The announcement lists LMCache 0.5.8rc1 or later, vLLM 0.11.1 or later or SGLang 0.5.13 or later, and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes. The getting-started guide independently confirms the LMCache and SGLang prerequisites and requires the same LMCache version in the server and engine image. These are feature minimums, not a recommendation to pin a prerelease indefinitely; check current authoritative component documentation before deployment.
For the RDMA path, GPU hosts must support EFA RDMA write and sit in the same Availability Zone as the cache. The skill says to plan one cache per Availability Zone and excludes SageMaker, serverless caches, replicas, snapshots, Multi-AZ and Global Datastore from its supported setup. Its plain-TCP functional-test path should not be mistaken for proof that the production RDMA path works.
The cache is ephemeral. The documentation says deleting the cluster permanently removes its data and that it cannot be backed up; the skill also warns that node replacement and patching empty the cache. Builders should explicitly decide whether a cache-load failure causes recomputation or a failed request rather than relying on an engine-version default.
Estimate useful reuse before paying for nodes
AWS’s workflow uses KVTrace to record cache events and replay them offline against candidate shared-cache sizes. According to the announcement, the recorder writes block hashes and token counts, not prompt text or token IDs. Because hashes are derived from token content, AWS recommends the --anonymize-key-file option for sensitive prompts. A trace still needs an approved collection and handling policy.
The published savings walkthrough uses an example fleet and scenarios when the hit rate is unknown; its displayed KVTrace report uses synthetic demo traffic. Neither is an AccessAllGPT measurement or a universal return on investment. AWS’s headline cost and latency improvement claims are vendor-reported, and we did not review or reproduce the linked benchmark study for this briefing.
The documentation warns that cache.i8ge.48xlarge nodes are billed per hour until the cluster is deleted. We are not quoting a regional hourly rate or a model API token price. Include the cache and operational overhead in the business case, then ask whether recovered GPU time will actually reduce capacity or serve more paid-for work. A faster first token on an unchanged idle fleet is not automatically a lower bill.
A useful first evaluation checks correctness as well as speed
Start with a representative, non-sensitive trace and an explicit no-resource-creation instruction. Compare local prefix caching and current routing with the predicted shared-tier benefit before approving any cluster. If the projected savings do not cover its cost, the skill’s stated workflow says to stop.
For an approved pilot, clear local GPU and LMCache L1 state, repeat known prompts, confirm L2 token-hit counters increase and compare outputs against a checked reference. AWS warns that layout mismatches can produce healthy hit counters and wrong answers. Align model/version, tensor-parallel layout, KV data type and hash settings across participating servers.
Tenant design also matters: AWS warns that shared-prefix timing can reveal whether another tenant sent the same text. The skill recommends separate caches where required and distinct cache salts for tenants sharing one. These are documented safeguards, not isolation guarantees we tested. Evaluate cold-cache recovery and tenant separation before moving production traffic.
Shared KV caching: a pilot decision card
Proposed operator checks, not hands-on results.
Entries stay in this browser tab and are not submitted to AccessAllGPT. Blank responses are copied as [Unresolved].
Repeated long prefixes, cross-replica recomputation and measured reuse; compare against local caching and session affinity first.
Compatible engine and LMCache builds, EFA RDMA write, same-AZ placement and consistent KV layouts; document cold-cache behavior.
Use a representative trace, subtract cache and operating costs, explain how capacity is saved, and verify known outputs after clearing local tiers.
Primary sources
Browse the publication-wide evidence index →
- Cut self-hosted LLM cost and latency with the distributed KV caching agent skillAmazon Web Services · Reviewed: Publication timestamp; skill introduction; fit checks; prerequisites; four workflow examples; KVTrace recording and synthetic report; validation and tenant cautions · Retrieved · Supports: Announcement date, intended agents and workloads, published component minimums, vendor-described workflow and explicitly illustrative economics. Does not establish independently measured savings.
- Distributed KV caching skill — public SKILL.mdAWS on GitHub · Reviewed: Overview; applicability and exclusions; transports; typical workflow; rules; unsupported platforms; tenant isolation · Retrieved · Supports: Public skill availability, LMCache architecture, same-AZ and ephemeral-data restrictions, unsupported services, fit-first cost assessment and tenant guidance. Mutable repository document, not a tested deployment.
- Getting started with distributed KV cachingAmazon Web Services documentation · Reviewed: Hourly billing warning; prerequisites; cluster creation; client hosts; engine connection; cache-use verification · Retrieved · Supports: Billable node lifetime, placement and EFA prerequisites, engine connection and version requirements, data-loss warning and L2 validation procedure.
Limitations
No AWS resources provisioned, skill installed, workload trace recorded, authenticated entitlement checked or inference run performed. No independent benchmark or cost measurement. The linked benchmark study was not reviewed for this briefing; headline improvement claims remain vendor-reported. Component and repository documents can change, and no exhaustive regional availability or price survey was performed.
Disclosures
Public-source reporting and AccessAllGPT analysis, not a sponsored test or private-access claim. AWS did not review or endorse this briefing. AccessAllGPT Research is operated by NeuralArc and is independent of AWS; publication-wide relationships appear on the disclosures page.



