Cloudflare's disclosure of a cross-request data leak in its Workers AI deployment of @cf/zai-org/glm-5.3 reveals a critical truth about modern AI infrastructure: in multi-tenant LLM serving, memory management bugs are security vulnerabilities. Between August 27 and September 9, 2026, 155 responses generated for prompts longer than 681,280 tokens returned fragments of other customers' sessions because a context-window mismatch collided with an error-handling flaw in the open-source SGLang inference engine.
The Performance Shortcut Behind Shared KV Caches
To understand why one user's prompt can bleed into another user's completion, we have to look at how inference engines manage GPU memory. When a large language model processes a prompt, it computes Key and Value (KV) tensors for every token so it does not have to recompute attention across the entire sequence during token-by-token generation.
In high-throughput engines like SGLang and vLLM, the KV cache resides in a shared pool of fixed-size GPU memory pages. When a request arrives, the scheduler allocates free pages to hold its context; when the request finishes, those pages are returned to the pool. Crucially, inference engines do not zero out GPU memory pages between requests. Zeroing gigabytes of high-bandwidth memory on every turn would introduce severe latency penalties. Instead, engines rely on a strict contract: a newly assigned page is always overwritten during the prefill phase before the decode phase ever reads from it.
That optimization is safe only as long as the engine's page-ownership tracking and state transitions are airtight. If a request bypasses the write step but still receives page pointers for reading, the model does not crash—it simply attends to whatever residual KV states were left behind by the previous tenant.
How Disaggregated Serving Amplified a Configuration Mismatch
Cloudflare served glm-5.3 using SGLang in disaggregated prefill/decode mode. Rather than running the entire request on a single GPU node, disaggregated serving splits the workload across two specialized pools:
- Prefill workers ingest the full user prompt in parallel, compute the initial KV cache pages, and transfer them over RDMA using the Mooncake transfer engine.
- Decode workers receive those populated KV pages and generate output tokens sequentially.
The incident required two overlapping failures to trigger. First came a configuration drift. Cloudflare configured glm-5.3 with its advertised 1,048,576-token context window. However, SGLang calculates its KV cache pool size dynamically at startup based on the GPU memory remaining after loading model weights. On the prefill GPUs, that pool only had capacity for 681,280 tokens. Because no startup assertion forced the physical KV pool to be at least as large as the configured maximum context length, requests between 681,281 and 1,048,576 tokens were accepted by the API gateway but could not fit into prefill memory.
Second came the SGLang state-machine bug, originally filed upstream as issue 30233 on July 6, 2026. In single-node mode, an over-capacity request immediately returned an error. In disaggregated mode, when a prompt exceeded the 681,280-token prefill pool, SGLang marked the request as rejected and replaced the oversized prompt with a single-token placeholder so the pipeline could drain cleanly. However, the admission queues coordinating prefill and decode checked only whether a request was marked finished, ignoring the rejected state.
As a result, the rejected request remained alive across the handoff. Prefill processed only the 1-token placeholder, reported the RDMA transfer complete, and handed the request off to the decode node. The decode worker then generated tokens by reading across allocated KV cache pages that had never been overwritten for that request—exposing raw conversational context from earlier users.
Why HTTP 200 Benchmarks Missed the Leak
One of the most instructive details in Cloudflare's postmortem is that engineers knew about the 681,280-token pool limit before launch. They expected SGLang to reject oversized prompts with an error, and when their pre-launch long-context benchmarks returned HTTP 200 status codes with streamed token bodies and no error frames, they assumed the engine was handling the boundary gracefully.
In reality, the benchmark harness was receiving HTTP 200 responses filled with hallucinated or leaked continuation text generated from uninitialized KV pages. Because the test suite only validated transport-level success (status == 200) and streaming latency rather than semantic coherence or canary token retrieval, the silent failure passed right through staging.
After security researcher Gadi Evron of Knostic reported the cross-tenant leak through HackerOne on September 8, 2026, Cloudflare reproduced the bug on September 9, took the model offline within 40 minutes of declaring the incident, and restored service the same day. They aligned the glm-5.3 deployment with their glm-5.2 configuration—expanding the prefill KV pool to roughly 2.04 million tokens—and collaborated with SGLang maintainers on upstream fixes (including PR 38704 and releases through v0.5.22) that abort rejected requests at admission before any KV allocation or bootstrap occurs.
Hardening Self-Hosted Inference Stacks
For engineering teams operating SGLang, vLLM, or TensorRT-LLM in multi-tenant environments, this incident proves that LLM serving security cannot be separated from GPU memory lifecycle management. Operators should implement three concrete safeguards:
def verify_kv_capacity(configured_max_context: int, prefill_kv_pool_tokens: int, decode_kv_pool_tokens: int) -> None:
"""Startup invariant check for disaggregated inference nodes."""
if prefill_kv_pool_tokens < configured_max_context:
raise RuntimeError(
f"Prefill KV pool ({prefill_kv_pool_tokens}) is smaller than "
f"configured context length ({configured_max_context}). Aborting startup."
)
if decode_kv_pool_tokens < configured_max_context:
raise RuntimeError(
f"Decode KV pool ({decode_kv_pool_tokens}) cannot hold full context."
)
- Enforce startup capacity assertions: Never let an inference container join a load balancer if its dynamically allocated KV cache pool on either prefill or decode nodes is smaller than
--context-length. - Upgrade disaggregated engines immediately: Teams running SGLang in disaggregated prefill/decode mode must upgrade to
v0.5.22or later so rejected requests are retired at admission rather than drained via placeholder tokens. - Test semantic coherence at boundary lengths: Long-context load tests must embed unique canary strings near the end of maximum-length prompts and verify their exact retrieval in the completion. In LLM infrastructure, an HTTP 200 status code only proves the socket stayed open—not that the memory boundary held.
Loading comments…