Someplace in a Singapore knowledge middle, a financial institution is paying for eight H100s that spend a lot of the evening ready. The cluster was purchased for good causes (discomfort with buyer paperwork leaving the constructing, a method group’s aversion to lock-in), so the financial institution secured its personal sovereign compute. Now the finance group is asking why a machine that prices extra per hour than a senior engineer runs at a fraction of its capability. That is the GPU hangover. During the last two years, enterprises rushed to lock in non-public clusters and reserved cloud nodes to construct AI they may management. The {hardware} arrived; the utilization didn’t. The explanation isn’t dangerous planning. It’s a mismatch between how commonplace fashions generate textual content and the way enterprises truly use them, and textual content diffusion is probably the most attention-grabbing candidate for closing the hole. It’s additionally probably the most oversold, and the oversell hides in which workloads it truly helps.
Begin with the physics. A normal autoregressive mannequin, from the Llama, Mistral, or GPT households, as an illustration, generates one token at a time. The weights by no means change and by no means depart the cardboard; they sit within the GPU’s high-bandwidth reminiscence the entire time. The bottleneck is one degree down. Arithmetic occurs solely within the chip’s tiny pool of on-chip reminiscence, which is nowhere close to large enough to carry a multibillion-parameter mannequin. So for each single token, the total set of weights needs to be streamed out of that fundamental reminiscence and thru the compute items once more—rereading the mannequin from the cardboard’s personal reminiscence into the cardboard’s calculators, as soon as per token, as a result of the calculators can not maintain it resident. The mathematics finishes nearly immediately and the items then idle, ready for the following slice of weights. Measured as arithmetic depth, operations per byte moved, this sits close to 1 at batch measurement one, whereas trendy GPUs are constructed for intensities within the a whole bunch. The chip is starved, bottlenecked not by a scarcity of compute however by the pace of the feed. The escape hatch is batching: Learn the weights as soon as and use them to compute the following token for a whole bunch of requests on the identical time, amortizing that one costly learn throughout a whole bunch of tokens of helpful work. On the identical {hardware}, small versus giant batches can swing value per token 10- to 30-fold, which is why public APIs, working monumental batches throughout 1000’s of customers, are low cost.
All the pieces hinges on whether or not you’ll be able to accumulate concurrent work. An in a single day queue of 1,000,000 paperwork is trivially batchable, as a result of no person’s ready. However when a single request should return in underneath a second, say a developer’s code completion or an onboarding test whereas the shopper stands on the counter, you’ve spent your latency price range and may’t wait to fill a batch. The primary type of workload will not be actually memory-bound; you batch your method out of it. The second variety is, and no quantity of complete quantity rescues it. And there’s an extra subtlety: Producing tokens is memory-bound, however studying the immediate is already compute-bound, for the reason that enter is processed in parallel. Doc extraction is usually studying, lengthy enter and quick output, so even a normal mannequin spends a lot of that job within the regime the place it was by no means starved within the first place.
Diffusion assaults precisely the half that’s starved. Borrowing its mechanism from picture era, it begins with a block of masked or noisy tokens and refines the entire block in parallel over a number of denoising passes, much less like a typewriter and extra like an editor revising a full draft directly. As a result of every go does actual arithmetic throughout the entire block, it’s compute-bound even at batch measurement one. The place autoregressive depth sits close to 1, a comparable diffusion mannequin’s lands within the a whole bunch. It saturates the compute you already pay for with out the concurrency you don’t have. The numbers are actual. Inception Labs’ Mercury reported over 1,100 tokens per second on H100s for code era, and the 2026 Mercury 2 launch reported roughly 1,000 tokens per second on Blackwell at low latency. Google confirmed the paradigm at frontier scale with Gemini Diffusion, and open supply LLaDA confirmed diffusion fashions observe autoregressive-like scaling legal guidelines. These are early however actual: Mercury 2 is commercially obtainable, Gemini Diffusion is in enterprise preview with common availability anticipated later in 2026, and the open fashions are maturing quick, at the same time as autoregressive programs nonetheless dominate on tooling and ecosystem quite than any theoretical ceiling. So the headline is true in a single particular place: for a latency-bound, single-stream request, diffusion can run an order of magnitude sooner, as a result of the autoregressive mannequin is caught memory-bound and can’t be batched out of it. However saturating the GPU is an engineering metric, and you’ll saturate a chip doing ineffective work. The actual query is what it prices to provide a helpful token, and on which workloads.
Earlier than declaring a winner, a good comparability has to account for what autoregressive serving can already do. Speculative decoding and its descendants, Medusa and EAGLE, use a small draft mannequin to suggest a number of tokens that the primary mannequin verifies in a single go, giving roughly two- to four-fold single-stream speedups with no change in high quality. Combination-of-experts fashions assault the identical wall from one other path, activating solely a fraction of their weights per token and so transferring much less reminiscence per token generated. The query is due to this fact not autoregressive versus diffusion within the summary; it’s whether or not diffusion’s structural parallelism beats a speculatively decoded mannequin’s incremental achieve on the workload you even have. For a decent single-stream latency goal, diffusion’s edge is giant and sturdy. For offline batch, neither trick issues a lot, as a result of batching already pushes each architectures into compute-bound territory. Any framing that ignores speculative decoding is promoting a false binary.
Whichever trick you attain for, the economics cut back to a single identification:
Efficient value per token = node value per hour ÷ (throughput × utilization)
A public API is priced per token, concurrency impartial, with no idle penalty. Owned compute is priced per hour, and its per-token value is derived from how a lot you push by, so throughput and utilization are the one levers, and diffusion strikes the primary one decisively however solely the place batching is unavailable. The costs make the stakes concrete. A reserved AWS p5.48xlarge, eight H100s, lists close to $55 an hour on demand, and one-year financial savings plans lower that by roughly 40 %, to about $33 an hour. Towards an affordable commodity API, a small mannequin underneath a greenback per million tokens, owned compute loses on pure value no matter structure; a $33-an-hour field, nevertheless properly used, can’t beat a token you’ll be able to lease for 40 cents. Diffusion’s financial win seems in solely two conditions: when the token you’d in any other case purchase is dear, frontier or reasoning output at $5 to $15 per million, the place a saturated owned node comfortably undercuts the API, or when the information can’t go to an exterior API in any respect, so the comparability turns into owned diffusion versus owned autoregressive. Most regulated enterprises stay in that second case.
Nowhere is the excellence clearer than within the financial institution’s personal doc operation, which has two faces that look alike and behave like opposites. The in a single day batch, thousands and thousands of KYC packets, letters of credit score, and mortgage information parsed into JSON whereas nobody waits, is the best attainable workload to batch. With steady batching, a normal mannequin runs at a number of thousand tokens per second and clears the queue on a single node; diffusion is considerably sooner and finishes the window sooner, however each match on one field at an identical value. If this had been the entire workload, switching architectures can be arduous to justify, as a result of autoregressive batching has already solved a lot of the downside, and this job is usually prefill anyway, its enter tokens dwarfing the JSON output an API would invoice for. The actual-time path inverts the conclusion solely. A relationship supervisor onboarding a buyer wants the paperwork parsed in underneath a second whereas the shopper waits; an officer clearing a letter of credit score wants the reply now; an agentic circulate is blocked on a single doc earlier than it will possibly proceed. These requests arrive separately, every with a tough latency price range, so you’ll be able to’t batch them, as a result of batching trades latency for throughput and there may be none to commerce. A big autoregressive mannequin in single-stream decode emits solely tens of tokens per second, so a number of hundred tokens of output take a number of seconds, and speculative decoding helps however doesn’t attain interactive pace, whereas diffusion returns the identical report in properly underneath a second. The price exhibits up as node rely, and now it’s appropriately attributed: to carry a subsecond goal with the autoregressive mannequin you should maintain batches tiny, so every node serves solely a handful of concurrent real-time requests and assembly peak demand means overprovisioning throughout many nodes, whereas diffusion clears every request quick sufficient that one node absorbs much more low-latency visitors and suits the identical service degree on a fraction of the fleet. The financial savings are actual, and so they come from the latency constraint defeating batching, not from low concurrency within the summary.
The lesson of these two jobs generalizes right into a routing rule sharper than the same old recommendation of customer-facing on APIs and inner on owned compute. The actual take a look at has two axes: whether or not the work will be batched, that means it’s offline-tolerant quite than latency-bound and serial, and what every token is value. Latency-bound, decode-heavy, low-value era akin to code completion, real-time extraction, and the chatter of agentic workflows is the diffusion candy spot, the place batching is unavailable, the standard hole is tolerable, and a quick owned node beats each an overprovisioned autoregressive fleet and an costly API. Excessive-value reasoning, the place a flawed reply is expensive, stays on frontier autoregressive fashions. And offline batch of any worth density goes to no matter you already run properly, as a result of batching has already made it environment friendly.
That self-discipline issues as a result of diffusion carries actual constraints. High quality isn’t free: Diffusion trades some accuracy for pace, touchdown round 85% to 95% of sturdy autoregressive baselines, aggressive on structured output however trailing by 5% to fifteen% on arduous reasoning, on vendor and secondary figures that deserve impartial verification towards your individual knowledge. That’s advantageous for discipline extraction and never advantageous for credit score selections, so any critical deployment budgets a fallback for outputs that miss a confidence threshold and folds its value again into the efficient price. Being compute-bound is itself a price, since diffusion earns its excessive depth partly by doing extra complete work per helpful token, which is why the metric that issues is all the time tokens per greenback at an appropriate high quality bar and by no means utilization by itself. The baseline can also be transferring: speculative decoding, higher schedulers, and mixture-of-experts fashions maintain narrowing the hole with no mannequin swap, so diffusion has to beat a transferring goal quite than the naive one. And the tooling is early, with open-source diffusion serving in 2026 sitting roughly the place open-source autoregressive serving did in early 2024, purposeful and bettering quick however quick on the mature inference stacks groups take as a right with vLLM or TensorRT-LLM. Each conclusion right here additionally strikes with two costs you don’t totally management, the API price you examine towards and the {hardware} price you negotiated, so it’s value relationship your assumptions and revisiting them.
The hangover, ultimately, will not be that enterprises purchased the flawed {hardware}. Many purchased it for causes like sovereignty, knowledge management, the avoidance of lock-in that don’t have anything to do with token economics and gained’t go away. They purchased it anticipating it to behave like a public cloud, then ran it at a concurrency that cloud economics rely on and that their most respected inner workloads, the latency-bound ones, can by no means attain. Textual content diffusion will not be a strategy to beat the API, nor a blanket improve for every part an enterprise runs. It’s a exact instrument for a exact hole, the latency-bound, decode-heavy, sovereignty-constrained work the place batching is inconceivable and an autoregressive mannequin leaves a node each starved and overprovisioned. For the copilots, the real-time checks, and the agentic steps that should reply now, it turns that node from a responsible line merchandise right into a saturated asset, on a fraction of the containers the choice would wish. That’s a narrower declare than rescuing your {hardware} ROI, and a much more sturdy one. The way forward for enterprise AI is the correct structure, on the correct {hardware}, carrying the correct tokens, and understanding which tokens these are is the half no vendor will promote you.
Sources for additional studying
Inception Labs, “Mercury: Extremely-Quick Language Fashions Based mostly on Diffusion” (arXiv:2506.17298) and Mercury 2 launch protection, February 2026
“Consistency Diffusion Language Fashions” (arXiv:2511.19269) on the arithmetic depth of autoregressive versus diffusion decoding throughout batch sizes
Baseten’s “A information to LLM inference and efficiency” on the reminiscence wall, batching, and the prefill versus decode distinction
Leviathan et al., “Quick Inference from Transformers through Speculative Decoding” (2023), with Medusa and EAGLE; AWS EC2 P5 pricing pages and 2025 P5 savings-plan bulletins
LLaDA2.0 (Bie et al., 2025) on the scaling conduct of diffusion language fashions.
Be aware: Throughput figures are engineering approximations for a 70B-class mannequin; substitute your individual measured numbers, at your individual batch sizes and sequence lengths, earlier than any procurement choice.
