Research · AMD ROCm Blog
UltraQuant Speeds Up Busy Long Sessions; MTP Helps Only Up to a Point
In AMD's experiment on eight MI355X GPUs, UltraQuant delivered 29% higher generation throughput than the strongest tested eight-bit cache configuration during a Qwen3.8 long-session agent replay with 32 concurrent requests. MTP, useful at lower concurrency, made performance worse at that peak.
What this means for a buyer
The study offers a specific configuration choice for large local agent systems on MI355X. Cache compression and MTP address different problems; the setting with the lowest latency under a modest request stream may become unsuitable when many long sessions run together.
Compressing weights and compressing a conversation are different
AMD's September 25, 2026 publication studies UltraQuant with Qwen3.8-2.4T-A95B and MXFP4 weights. MXFP4 reduces the representation of model weights, while UltraQuant separately stores the KV cache at four bits. That cache holds intermediate data for text already processed. In long agent workflows, it is a distinct memory expense: conversation history, tool results, and session continuation add context even when the next answer is short.
The studied Qwen3.8 has 2.4 trillion parameters, approximately 95 billion active per token, and a 262K window. The active portion of a mixture-of-experts model, which selects experts, is not equivalent to all the weights that must be placed in memory. Another important detail is that only 23 of its 92 layers keep a conventional KV cache; the others use lightweight recurrent state. Results therefore cannot be transferred mechanically to every model: the architecture's share eligible for this compression is specific.
Why four bits actually reduce work
UltraQuant stores a group of 32 cached values in 17 bytes, compared with 32 bytes for the eight-bit format. The generation benefit is straightforward: less data needs to be read from memory. However, that saving can disappear if each value requires expensive reconstruction before computation. In the described MI355X path, FP4 enters matrix operations directly, and scaling is incorporated into the corresponding instruction. The compact representation therefore survives into the compute kernel.
Before compression, a Walsh–Hadamard transform spreads unusually large values across channels. This helps represent the data on a four-bit numerical grid. For a buyer, the complete chain matters: preparing the distribution, storing it in the chosen format, and hardware support for that format. A claim of a four-bit cache alone is insufficient to predict the same benefit on another card. UltraQuant's specificity here comes from the MI355X native computational path, beyond a smaller storage footprint.
What AMD actually compared
The authors replayed a SemiAnalysis InferenceX agent trace with long shared prefixes, inputs ranging from tens to hundreds of thousands of tokens, short outputs, and many turns. They used eight MI355X GPUs, vLLM v0.19.2rc0 with the studied UltraQuant kernels, AITER 0.1.16.post3, and FlyDSL 0.2.0. The context was 262K, with prefix caching enabled. Prefix-cache hit rates stayed at 91–95% across configurations, and KV-cache offloading was not used.
At 32 concurrent requests, UltraQuant delivered aggregate throughput of 434 tokens per second with 58 ms inter-token latency. The compared eight-bit ROCm-FA path produced 338 tokens per second at 77 ms: 29% higher throughput and 24% lower latency for UltraQuant. AMD selected the stronger of two tested eight-bit paths at each load. At light load, the difference disappeared. This is a result for busy long-session traffic with substantial prefix reuse, rather than a speed prediction for one short chat.
Why MTP should not simply stay enabled
Multi-Token Prediction, or MTP, proposes several candidate tokens and verifies them together. Adding it to UltraQuant reduced inter-token latency by a further 12–22% at concurrency up to and including 16. At 32, both MTP configurations regressed. The authors attribute this to additional verification work: at lower load, fewer sequential steps are advantageous, while at the peak the verification cost becomes significant. This creates a concrete configuration choice based on expected concurrency rather than a universally beneficial switch.
The studied UltraQuant path also lacked a fused multi-token kernel. Candidates were processed as separate queries against the shared cache, without reusing one cache read across their positions. The eight-bit MTP path used AITER's fused kernel. The authors list fusion for UltraQuant and upstreaming the studied implementation into vLLM as future work. The measured experimental path consequently cannot be presented as an already available standard feature of every stable serving package.
The configuration choice supported by the results
AMD evaluated quality on GPQA-Diamond's 198 questions and a fixed 100-task SWE-bench Lite subset involving code fixes. Evaluations used temperature 1.0. Configuration results differed; the authors interpreted the gaps as consistent with sampling variation. Each SWE-bench configuration made one pass through the cohort. This is useful evidence for the studied path, but it does not establish equal accuracy across arbitrary repositories or kinds of multi-step task.
The conclusion concerns configuring a large local agent server on MI355X: consider UltraQuant for long overlapping sessions, and select MTP according to load. If the planned peak resembles the study's 32 concurrent requests, these results provide no basis for keeping MTP permanently enabled. At lower concurrency, it delivered an additional benefit. The interesting choice is the execution mode itself: reducing sequential steps and reducing memory reads help under different conditions, so settings should reflect the actual work stream.
What to check before choosing
- What share of requests starts new sessions without a shared cached prefix?
- Which code-fixing errors are unacceptable regardless of speed?
- How does switching MTP affect sessions already open in the chosen implementation?
- Does response time remain acceptable when another server task runs alongside it?
Buyer implications are our editorial interpretation. The publication does not establish the speed or compatibility of a particular catalogue system.