Research · NVIDIA Technical Blog
ReplaySSM: faster hybrid models start with fewer writes to memory
On October 12, 2026, NVIDIA described ReplaySSM, a way to store hybrid-model state using a checkpoint and recent inputs. Large request batches benefited in its tests, while a batch of one saw no speedup. That distinction helps buyers choose how a shared local server should operate.
What this means for a buyer
Buyers should connect acceleration to their team’s actual concurrent workload. ReplaySSM supports evaluating a specific execution-engine mode: its state handling, draft-token verification, and update precision. A model name alone does not establish these server characteristics or their value in use.
Fixed-size state can still overload memory
The NVIDIA and Dao AI Lab research examines hybrid models that interleave SSM and attention layers. An SSM, or state-space model, maintains fixed-size state during generation. An attention layer uses a KV cache that retains information about previous tokens and grows with context. Fixed-size state looks attractive, but it does not guarantee a fast next token.
Conventional SSM execution loads the full state, updates it, and writes it back at every step. When serving large batches, moving this data can constrain speed more than arithmetic. The authors demonstrate this bottleneck in hybrid models. To assess performance, it is therefore useful to know how much data the engine moves between successive token computations, even when the state itself does not grow.
A checkpoint replaces constant rewriting
ReplaySSM stores a state checkpoint alongside a small buffer of recent inputs. On a regular step, it appends the new input and computes the output from the buffer and checkpoint. It writes the fully updated state when the window fills. Producing the output is thereby separated from updating information needed by subsequent steps.
The main tests used a window of 16 records. The checkpoint is still read every step, but written once per window. The authors describe roughly halving the dominant state traffic. Outputs match conventional execution up to floating-point differences. This changes the pattern of memory operations, so its value depends on how strongly memory limits the particular workload.
Large batches benefit; a single request is a separate case
NVIDIA compared its ReplaySSM implementation in vLLM with vLLM’s own kernels using their best launch configuration. The tests covered Nemotron-3-Nano-4B and Qwen3.5-4B with BF16 weights on one H100, plus Nemotron-3-Super-120B-A12B and Qwen3.5-122B-A10B with NVFP4 weights on one Blackwell Ultra B300. Each model ran on one GPU without tensor parallelism.
With FP32 state, a window of 16, and batch size 256, the entire decode step was 1.20–1.47 times faster across the four models. At batch size 1, these tests showed no speedup because other latency sources dominated. This is a useful distinction for teams: a heavily shared server and a personal chat with occasional requests can benefit differently from the same optimization.
Drafts no longer need individual snapshots
Speculative decoding proposes several draft tokens, verifies them with the target model, and retains the accepted prefix. Conventional SSM state updates are difficult to reverse; the described workaround saves a snapshot after every draft token. ReplaySSM places inputs in a buffer and commits the accepted prefix by moving a pointer. This changes both verification and the memory needed for rollback.
On GSM8K prompts, using four draft tokens from each model’s MTP head, a window of 16, and FP32 state, the two 120B-class models achieved 1.85 and 2.10 times the throughput of vLLM’s speculative decoding at batch size 512. Each ran on one B300, with matched acceptance lengths. These results describe a particular serving configuration. Code-patch and document requests require measurements of the buyer’s own workload.
Four-bit reads need separate precision
State has another important property: rounding error can feed into the next update. ReplaySSM allows different precisions for reads that produce outputs and for the recurrent checkpoint. A low-precision copy serves reads, while later updates use a separate, higher-precision copy. Weight precision, attention-cache precision, and state precision therefore need separate consideration.
On GPQA-Diamond, with standard decoding and a window of 16, Nemotron-3-Super-NVFP4 achieved baseline accuracy of 79.3% with FP32 state. A shared NVFP4 checkpoint with stochastic rounding lost 12.5 percentage points. NVFP4 reads with a separate FP16 recurrent checkpoint and stochastic rounding lost 1.4 points. For Nano-4B, the latter configuration still lost 6.7 points: quality recovery varies by model. For a busy shared server, the specific choice is a ReplaySSM mode with separate state precisions, evaluated on the selected model and its typical response lengths.
What to check before choosing
- Which SSM mode is actually active in the engine, and how does diagnostics show it?
- What is the distribution of actual batch sizes, including quiet periods?
- How much memory do draft-state snapshots occupy in the current mode at peak load?
- Which long responses could reveal quality loss after changing state precision?
Buyer implications are our editorial interpretation. The publication does not establish the speed or compatibility of a particular catalogue system.