Practice · vLLM
Why vLLM Sends Conversation Cache Back to the Input Worker
vLLM describes a way to send retained cache from the generation worker back to the worker handling the next request. In a team’s long conversations, this changes the choice between transferring data and repeating computation. Cache routing and the interconnect may deserve attention before another accelerator in an already disaggregated service.
What this means for a buyer
A local service with separate input-processing and generation stages gains a specific option for improving conversation continuations. Its usefulness should be assessed through delays on subsequent turns and the actual cache-transfer path, rather than the speed of unrelated short answers.
The Next Question Brings Old Text Back
In its October 12, 2026 article, vLLM examines bidirectional KV-cache transfer for multi-turn conversations. The key change is that the worker processing incoming text can retrieve intermediate data retained by the answer-generation worker. For a team, this matters because delays when continuing a conversation may originate in processing its history again, rather than in the model’s size.
The source begins with a characteristic of these requests: the client resends conversation history and tool results. Imagine an assistant discussing a bug fix, receiving command output, and continuing its analysis; this is a hypothetical example. The user sees only a few added lines. The server, however, receives familiar conversation content again. Previously completed computation becomes a resource worth reusing.
One-Way Transfer Covers Only Half the Cycle
In a disaggregated arrangement, prefill processes incoming text while decode generates the continuation. In the conventional design described by the authors, cache moves from the first stage to the second. That route serves the current answer. Afterward, however, cache retained by decode is useful at the stage preparing the next request. The transfer direction no longer matches the next stage that needs the data.
Reverse transfer closes that loop: prefill reuses data that decode retained after answering. The benefit therefore concerns a sequence of related requests. A demonstration using unrelated short questions says little about this advantage. Answer length alone does not explain the wait for the first new fragment, either: the system must first handle the context that arrived with the request.
The Cache Bypasses the Router
In this architecture, the router stores metadata about cache blocks, while the tensors themselves travel directly between workers. Metadata can be understood as information about retained blocks; tensors are arrays containing intermediate computation. This distinction matters when choosing a server arrangement: the central component directs work, but the entire volume of transferred cache does not pass through it.
The authors use NIXL READ in both directions. Buyers need not study its programming interface to understand the infrastructure implication: the path between workers matters. Faster computation on one worker does not establish whether transferring data to another is worthwhile. A proposal for this arrangement should therefore describe the interconnect as well as the accelerators.
Transfer the Result or Compute It Again?
The authors also identify a configurable recomputation threshold that affects whether remote cache is retrieved. Behind this setting lies a straightforward choice: move already computed data or process the text again. Neither option can be declared best for every request in advance. The particular conversation history and transfer conditions matter; the threshold makes this choice part of the runtime configuration.
This suggests a useful question for the integrator: what happens with a short continuation versus a return to a long history? If the rule selects recomputation, the absence of a transfer does not by itself indicate a fault. Visibility into the selected path and its duration helps explain the delay instead of attributing every slow answer to insufficient computing power.
For a Team, the Second Request Matters
In vLLM’s experiment, Qwen3-32B time to first token at a 24K input fell from 778 to 460 ms. Prefill and decode used separate EC2 p5en.48xlarge nodes, each with eight H200 GPUs, connected by EFA; Qwen ran in bfloat16 with tensor parallel 8 on each node. Prefix caching was disabled on prefill to emulate cache loss. The reproduction recipe uses vLLM 0.29.0, NIXL 1.4.1 and a patch that returns cache metadata in streaming responses. This is a result for a specific multi-turn workload, rather than a promise of that improvement on one server. An owned cluster depends on its interconnect, pauses between turns and cache retention.
One focused comparison can assess this: run the same multi-turn session with and without reverse transfer, measuring the wait for the first fragment on continuation turns. Keep the model, history, and tool results identical. The purchasing implication is specific: for an already disaggregated team service, evaluate bidirectional caching before buying additional computing capacity. Its strength is preserving completed work between conversation turns.
What to check before choosing
- How does the router discover which worker holds the required blocks?
- Which recomputation threshold is configured for the team’s workload?
- How does a request continue when a required retained block is missing?
- How does continuation latency change when several sessions run concurrently?
Buyer implications are our editorial interpretation. The publication does not establish the speed or compatibility of a particular catalogue system.