The inference engine DeepSeek uses to serve its V4.1-Flash model, released on GitHub on 20 October 2026 under Apache 2.0. It began as a fork of vLLM and now runs one model on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The release covers offline inference only. The KV cache storage system, the prefill and decode disaggregation service, and the load balancers are described in a Q&A document but not included.

What is deepseek-recipe?

A companion Rust library with Python bindings, MIT licensed, that converts Chat Completions, Responses and Messages requests into one conversation type, renders the V4.1 prompt, tokenizes it, and parses the token stream back into each protocol. DeepSeek says it is the same code path as its own API, so third-party deployments get identical behaviour.

What do the 155 trillion tokens a day and $38 per GPU-hour figures imply?

With the published prices, DeepInfer's own throughput curves and the stated 1:1 prefill-to-decode GPU ratio, they leave two readings. At the decode operating point the Q&A names, the cluster is about 1,700 to 2,400 GPUs, 95 to 97 percent of input tokens are cache hits, and output is under 1 percent of tokens. If the 100 PiB cache belongs to that cluster alone, it is about 6,500 to 7,800 GPUs at 85 to 90 percent hits and 3 to 4 percent output, which needs decode to run at twice the stated concurrency. In both, cache hits are the large majority of tokens and at most half the revenue. In February 2025 DeepSeek reported a 56 percent hit rate and output at a fifth of tokens.

Why does the KV cache matter so much for DeepSeek's cost?

A cache-hit input token costs the customer $0.003 per million and costs DeepSeek a hash lookup and a read from SSD. A cache-miss token costs $0.15 per million and a trip through the prefill GPUs. Agent traffic resends the whole context every turn, so nearly all of its input tokens are hits. The persistent cache turns that traffic from a compute problem into a storage problem, and storage nodes are cheaper and fail less often than GPUs.

What is SWA bounded replay?

V4.1-Flash uses a 128-token sliding window in every layer. Rebuilding that window state exactly for a cached prefix would mean running the last 2,560 tokens through the 20 decoder layers. DeepInfer instead recomputes only the last 128 to 140 tokens and truncates attention to the replayed segment, which makes the recomputation cost constant and, by DeepSeek's account, nearly halves total prefill work. The resulting decoder state is an approximation, not the exact one. Measured on real code and prose, it raises perplexity on the true continuation by at most about 0.4 percent and changes the top token at no more than about 1 percent of positions. Those are ceilings, since the test had no run to subtract rounding noise.

Should I run DeepInfer myself?

Only with a large fleet. DeepSeek's smallest efficient unit is 64 prefill plus 64 decode GPUs, with about 14 TB of logical SSD cache per GPU and the storage cluster, load balancers and autoscaler built yourself. It also expects an RDMA network: its Engram lookups will not start on a node without an InfiniBand device, even with every GPU in one box. For smaller deployments the released code is still worth reading for the cache policy, the scheduler and the serving constants, and vLLM already serves the model on a wider range of hardware.

DeepInfer: The Engine Is Open. The Cache Is the Business.

The short version, without the jargon: DeepSeek runs one of the cheapest large-model APIs in the world. On 20 October it published the software that serves it, plus a document that says how much it serves (155 trillion tokens a day from one cluster) and how much that earns (\$38 per GPU-hour at its cheapest prices). I did the arithmetic. Those numbers only work if almost everything DeepSeek serves is text it has already processed before and kept on disk, so that the second, third and hundredth time a customer sends the same conversation it is a lookup rather than a computation. The software that does that lookup, and the storage system behind it, are exactly the parts DeepSeek described but did not release. The rest of this post is the detail.

TL;DR: DeepInfer is a squashed vLLM fork, four commits deep, that runs one model on two kinds of chip and ships offline inference only. The real disclosure is the deployment Q&A beside it. Its numbers leave two readings. Either a cluster of about 2,000 GPUs serves 155 trillion tokens a day with 95 to 97 percent of input tokens hitting the cache and output under 1 percent of tokens, or, if the 100 PiB cache is that cluster’s alone, about 7,000 GPUs do it at 85 to 90 percent hits with decode running much hotter than the Q&A says. Either way, cache hits are the bulk of what is served and at most half of what is earned. In February 2025 DeepSeek reported a 56 percent hit rate and output at a fifth of tokens. The cache’s hit policy and the model share the same constants, and replaying that policy on synthetic traffic shows where it wins (agents, long documents) and where it pays a small per-turn tax (short chat turns). The decoder shortcut that makes the policy cheap costs, on real code and prose, at most about 0.4 percent in perplexity and a different top token at no more than about 1 percent of positions. What was held back is the storage cluster, three load balancers and the autoscaler. That is the product.

Code: github.com/bassrehab/deepinfer-lab: the economics model, the cache policy simulator, a probe for the companion deepseek-recipe library, and the harness that measured the replay approximation on two B300s. Only the replay measurement needs a GPU; the rest runs on a laptop.

I cloned DeepInfer the morning it appeared and did the boring things first. Four commits on main. About 55,000 lines of Python and 6,000 of C++, with about a third of the Python files commented in Chinese. One supported model, DeepSeek-V4.1-Flash. Two supported chips, NVIDIA SM100 and Huawei Ascend 950. The README says “offline inference” and then lists what you would need to add to serve traffic with it: a KV cache system, prefill/decode disaggregation, and load balancing across replicas. In the engine those are literal stubs. The method that sets up prefill/decode separation is a one-line virtual function whose docstring, translated, reads “virtual function, initialise the data structures PD separation needs.”

Then I opened the 29 KB document in the doc/ folder, titled “Inference Deployment Q&A”, and stopped reading code for the rest of the day. It describes the whole production system, including the parts that were not released, with numbers: how big the storage servers are, how the four load balancers route, what the cache TTL is and what it costs in hit rate, how expert parallelism is rebalanced, and, in passing, how many tokens a cluster serves per day and what that earns per GPU-hour. The only comparable disclosure I know of is DeepSeek’s own, from February 2025, and I come back to it below. So this post is not a tour of the repo. It is an attempt to work out what the disclosed numbers mean, and to check the design they describe against what the code and the model actually do.

What shipped, and what was described

The Q&A’s architecture diagram has a request edge, an entrypoint, three load balancers, three kinds of engine, a transfer path between them and a storage cluster. The edge and the engines are public.

edgedeepseek-recipeprotocol conversion, prompt rendering, tokenization, stream parsing. Released, MIT.

entrypointPrefix match against the cache directoryhash-chain match on metadata in the CHECK role. The hit length decides routing, so this step runs before anything else. Described, not released.

↓↓↓

balancerVision encoder LBroutes images at machine granularity to the GPU with the fewest patches in flight.

balancerPrefill LBfour context-length tiers, engines pull work by remaining compute quota, no prefix-affinity routing.

balancerDecode LBbalances request count and total KV length at once: short requests go to GPUs with few requests but a lot of KV, long ones to GPUs with little KV.

↓↓↓

engineVision encoder workerone GPU per instance. In the release.

enginePrefill engine + EPLBEP2 to 8, chunked prefill, SWA bounded replay, structured-output masks, expert rebalancing every 1,000 steps. In the release.

engineDecode engine + EPLBEP64 to 128, DSpark with confidence-scheduled verification, async scheduling, batch-invariant mode. In the release.

transferPrefill to decode handoff of block KV and SWA statethe engine has the hooks; the methods are empty virtual functions in the released llm_engine.py.

storageKV cache store on 3FS, ~100 PiB SSD, 10+2 erasure coding, LRU garbage collection per nodeplus the autoscaler, the engine grace period that forwards unfinished requests, and EP failover that re-sizes an instance around dead GPUs. Described, not released.

in the releasedescribed in the Q&A, not releasedstubbed in the released code

The serving system as the Q&A describes it. The engine, its kernels and the request edge are public. Everything that decides which GPU a request lands on, whether its prefix is already computed, and where that computation lives, is not. DeepSeek says those components are "tightly coupled to our internal infrastructure".

That split is not unusual. Every lab keeps its orchestration private. What is unusual is that the Q&A then explains the private parts well enough to reason about them, and gives two numbers that let you do more than reason.

The two numbers

Here is the sentence, from the introduction of the Q&A:

A single cluster currently serves 155 trillion tokens per day, with overall revenue of US\$38 per GPU-hour (at valley pricing).

“Valley pricing” is the off-peak API rate, which is half of peak. The pricing page for the Flash model, as of this week, is \$0.003 per million cache-hit input tokens, \$0.15 per million cache-miss input tokens and \$0.60 per million output tokens, off-peak. Those three prices differ by a factor of two hundred, so “155 trillion tokens” could mean almost any amount of money depending on the mix. That is what makes the second number useful. \$38 per GPU-hour is \$912 per GPU-day. Divide the daily revenue the mix implies by \$912 and you get a GPU count. Then ask whether that many GPUs could have computed the tokens that needed computing.

The Q&A gives you what you need for the second half. Prefill and decode run on separate GPU pools in a ratio of “roughly 1:1”. Live throughput is “roughly 80 percent” of the offline benchmark. And the repo has the offline benchmark: two Pareto charts, prefill at 60,000 to 89,500 non-cached input tokens per second per GPU depending on step size, decode at 2,600 to 26,200 output tokens per second per GPU depending on how many requests share a rank. The Q&A puts production decode near 60 sequences per rank, around 14,500 tokens per second per GPU.

The 1:1 ratio is the constraint that does the work. If prefill GPUs are as numerous as decode GPUs, and each pool runs at its benchmark rate, then the cache-miss tokens computed per day and the output tokens generated per day are tied together by the two throughputs. Pick a share of output tokens and the cache-hit rate is no longer free. It is whatever value keeps the two pools balanced. From there you get the revenue, then the GPU count, then whether that count could do the work.

The Q&A also says how big the cache is: about 100 PiB of SSD, and about 14.4 TB of logical cache per online GPU. Divide one by the other and you get a second GPU count that has nothing to do with revenue. It is 6,500 if the 100 PiB is physical capacity and 7,800 if it is logical. The instrument below does all of this live.

Instrument / What the disclosed numbers imply

155 trillion tokens a day at $38 a GPU-hour. What mix of tokens makes that true?

DeepSeek published the daily token volume, the revenue per GPU-hour, the API prices, the throughput curves of its own engine, and the fact that prefill and decode GPUs run roughly 1:1. Those five things pin down the token mix more tightly than you would expect. Pick a share of output tokens and an operating point on DeepInfer's decode chart; the instrument solves for the cache-hit rate that keeps prefill and decode balanced, then checks whether the GPU count the revenue implies can actually compute the misses and outputs in a day. A third disclosure, about 14.4 TB of cache per GPU on a store of about 100 PiB, says how many GPUs sit behind the cache. The shaded horizontal band is that count. Watch where the lines cross it.

Output tokens, share of all tokensiOf the 155T tokens a day, the fraction the model generated. The rest are input tokens, cached or not.

Decode operating pointiPoints from DeepInfer's decode Pareto chart (EP64/DP64, 131K context). The Q&A puts production near 60 sequences per rank.

Prefill operating pointiPoints from DeepInfer's prefill Pareto chart (EP4/DP4, 131K context, non-cached input tokens per second per GPU).

Online efficiencyiHow much of the offline benchmark the live, load-balanced service achieves. The Q&A says roughly 80%.

Persisted bytes per tokeniThe tech report gives 890 B/token of global KV in HBM. The SSD copy is assumed the same size before the 1.2x erasure-coding overhead.

Cache-hit rate of input tokens

GPUs implied by $38 / GPU-hour

GPUs needed at benchmark rate

Days of computed tokens 14.4 TB per GPU holds

Does it close?

tokens

revenue

cache-hit input, $0.003 / Mcache-miss input, $0.15 / Moutput, $0.60 / M

GPUs implied by revenue GPUs needed at this operating point feasible: capacity fits inside the revenue count GPUs behind a 100 PiB store at 14.4 TB each

token class	tokens / day	share of tokens	share of revenue

How this is computed

Let T be 155 trillion tokens a day, o the output share and h the cache-hit rate of input tokens. Then outputs are O = oT, hits H = h(1 - o)T and misses M = (1 - h)(1 - o)T. Daily revenue at off-peak prices is 0.003H + 0.15M + 0.60O per million tokens, and dividing by $38 × 24 gives the GPU count the revenue implies.

Prefill GPUs must compute the misses: M / (prefill tok/s × 86,400 × efficiency). Decode GPUs must produce the outputs the same way. The Q&A says the two pools run roughly 1:1, which fixes h for any o: 1 - h = o × prefill / (decode × (1 - o)). The sum of the two pools, at benchmark throughput all day, is the capacity count. If it is below the revenue count the mix is feasible, with the ratio being the average utilization the fleet would need.

Storage: the tokens written to the cache each day are the misses plus the outputs. Holding them for the published TTL at the chosen bytes per token, times 1.2 for the 10+2 erasure coding on 3FS, gives the PiB figure, which the verdict compares with the roughly 100 PiB DeepSeek says it runs.

Store band: the Q&A gives about 100 PiB of SSD and about 14.4 TB of logical cache per online GPU. Dividing gives 6,516 GPUs if the 100 PiB is physical (logical is 100 / 1.2) and 7,819 if it is already logical. The retention readout divides 14.4 TB by the bytes each revenue-implied GPU's share of misses and outputs adds per day.

Not modelled: vision-encoder GPUs, storage nodes, the diurnal shape of traffic, peak-hour pricing (double the off-peak rates), and whatever a "cluster" means in the original sentence. The code is in deepinfer-lab.

(Standalone and embeddable version: What the disclosed numbers imply.)

Here is what the sweep says.

At the decode point the Q&A names, the cluster is small and almost every input token is a hit. At 60 requests per rank, about 14,500 output tokens per second per GPU, revenue and compute only agree when output is between 0.25 and 0.75 percent of tokens. The hit rate is then 95 to 99 percent and the cluster is roughly 1,000 to 2,400 GPUs. At the top of that range the GPUs would have to run at benchmark rate around the clock.
The cache says the cluster is bigger. 100 PiB at 14.4 TB per GPU is 6,500 to 7,800 GPUs, three to six times the first answer.
Getting revenue up to that count takes a different mix and a hotter fleet. Output would have to be 3 to 4 percent of tokens, with hits at 85 to 90 percent. Compute only keeps up if decode runs at 128 or more requests per rank, twice what the Q&A names. Even then a seven-day TTL would need 150 to 190 PiB of SSD. A three-day TTL would need 60 to 80, so this only works if most of the traffic is on the three-day agent TTL.
The 14.4 TB ratio fits both. At every point in the sweep, 14.4 TB per GPU holds four to six days of that GPU’s computed tokens, which sits between the two published TTLs. The ratio doesn’t pick a cluster size. Only the 100 PiB total does.

So the disclosures leave two readings. In one, the 155 trillion tokens come from a cluster of about 2,000 GPUs running where the Q&A says production runs, and the 100 PiB store serves more than that one cluster. In the other, the store is this cluster’s alone, the cluster is about 7,000 GPUs, and its decode runs a lot hotter than described. I lean toward the first. The 60-per-rank figure is given as the production operating point, and the storage figures read like numbers for the whole system. But I can’t rule out the second, and the instrument shows both.

The caveats are real and I have tried to keep them visible. The model ignores vision-encoder GPUs, storage servers and network, so the true cost per token is higher than the GPU count suggests. It ignores the daily shape of traffic. “Utilization” here is the average the fleet would need over 24 hours, and DeepSeek flattens its curve with off-peak pricing and autoscaling, but not to a flat line. Both Pareto charts were measured at 131K tokens of context, and real traffic is a mix of lengths. And the model assumes the persisted bytes per token equal the 890-byte HBM figure from the technical report, which the report does not quite say. Move any of those and the numbers shift.

Then and now

What doesn’t change between the two readings is the shape, and the shape is easiest to see next to the last time DeepSeek published numbers like these. At the end of its open-source week in February 2025, DeepSeek posted a day of traffic for V3 and R1: 608 billion input tokens, of which 56.3 percent hit the on-disk cache, and 168 billion output tokens, on an average of 1,814 H800 GPUs.

February 2025V3 + R1, measured

tokens

44%34%22%

revenue

9%26%66%

56% of input hits the cache · 1,814 GPUs · 0.43B tokens per GPU a day · $12.91 per GPU-hour

October 2026, reading AV4.1-Flash, modelled

tokens

97%

revenue

29%42%30%

97% of input hits the cache · about 1,700 GPUs · 90B tokens per GPU a day · $38 per GPU-hour

October 2026, reading BV4.1-Flash, modelled

tokens

84%13%

revenue

44%50%

87% of input hits the cache · about 7,200 GPUs · 22B tokens per GPU a day · $38 per GPU-hour

cache-hit inputcache-miss inputoutput

Token mix and revenue mix, measured in February 2025 against modelled for October 2026. The 2025 revenue is at R1 list prices, the basis DeepSeek used for its own theoretical figure. The 2026 revenue is at V4.1-Flash off-peak prices. Reading A is 60 requests per rank and 0.5 percent output. Reading B is 128 requests per rank and 3.5 percent output. Both use 16K-token prefill steps.

Three things moved.

Output fell from a fifth of tokens to a few percent or less. This is what agent traffic looks like. Every turn resends the whole context and gets back a tool call or a short answer.
The hit rate went from 56 percent to somewhere between 87 and 97. Agents again. The context they resend is the context the cache already holds.
DeepSeek repriced hits to match. In 2025 a cache hit cost about a quarter of a miss. Now it costs a fiftieth. Pricing hits near zero rewards clients that keep their prefixes stable, which raises the hit rate, which is what makes the low price affordable.

The per-GPU numbers moved the most. In 2025 each GPU handled about 430 million tokens a day, and DeepSeek’s theoretical revenue at list prices came to about \$13 per GPU-hour. Now each GPU in the cluster handles somewhere between 20 and 90 billion tokens a day and earns \$38 an hour at the cheaper of two prices. The hardware changed too, from H800 to SM100, so not all of that is the cache. But a fifty- to two-hundred-fold jump in tokens per GPU doesn’t come from a faster chip. It comes from not computing most of the tokens.

I want to be plain about what this does and does not show. It does not show a margin. The 2025 post assumed a \$2 hourly lease per H800 and printed a theoretical cost-profit margin. The Q&A gives no cost at all. Chinese coverage of this release has put the gross margin at around 80 percent, but that is a third-party estimate. The storage cluster alone adds 14.4 TB of SSD per GPU, plus the servers and network around it, before you count power or the GPUs. What the numbers do show is which token DeepSeek’s business is built on. It is the one it never computes.

What a cache hit actually is

That sentence only means something if you know what a hit costs, so here is the path of a request whose prefix is already in the cache, as the Q&A describes it.

Entrypoint

kvcache-store

Prefill engine (host)

3FS

GPU

Entrypoint → kvcache-store1prefix match on metadata, hash chain over 128-token blocksserial. The hit length decides routing, so this is the one step that cannot be hidden.

Entrypoint → prefill engine2route: tier by context length (<16K, 16 to 64K, 64 to 256K, ≥256K), pull by remaining compute quotaany engine can hit any prefix, so there is no prefix-affinity routing to fight the balancer.

Prefill host → 3FS3fetch the hit prefix's KV blocks into host shared memorya separate process, asynchronous. Reads go to all 11 nodes of a 10+2 erasure group, and the first 10 answers decode.

Prefill host → GPU4swap in host to HBM over the RDMA NIC, one layer at a time, overlapped with the forward passthe CUDA copy engine is slow on fine-grained block copies, so the NIC does it and no SMs are spent.

GPU5encoder prefill over the uncached suffix, decoder replay of the last 140 tokens, SWA state stored at a 128-token boundary

Prefill engine → decode engine6hand block KV and SWA state to a decode engine (not drawn)the prefill-to-decode path. In the release this is an empty virtual function.

A cache hit, step by step, from the Q&A's "KV cache storage" section. Only step 1 is on the critical path by design. The fetch and the swap-in are hidden behind other work, and the recomputation in step 5 is bounded regardless of context length.

Three design choices in that path are the ones I would steal.

The cache is a storage cluster, not a tier of local memory. Most open serving stacks cache prefixes in the host memory of the machine that computed them, which means the router has to send the next turn to the same machine, which fights the load balancer. DeepSeek puts the block KV cache on a dedicated 3FS cluster so that “every prefill machine can hit any prefix”. The Q&A gives two further reasons: storage nodes autoscale independently of compute, and compute nodes “fail at a rate more than an order of magnitude higher than storage nodes”, so the cache survives GPU failures. The recommended storage server is 20 SSDs of 30.73 TB each, with at least 40 GB/s of read bandwidth and 5 GB/s of write, and enough of them in total to hold at least a day of data.

Retention is long, and its cost is measured. The TTL is three days for agent traffic and seven for chatbot traffic. Chatbot traffic has “a much heavier long tail of hits”, a small amount of cache read many times, and dropping its TTL to three days costs 0.5 to 1 percent of hit rate. At \$0.147 per million tokens of difference between a hit and a miss, that fraction of a percent is what pays for the extra four days of SSD.

Decode writes to the cache too. Decode is not capacity-bound, so there is no KV offload during generation, but every token the model produces is still written to disk “in order to improve the cache hit rate”. The next turn of an agent session starts with the previous output as prefix. If it were not cached, the first thing the next turn would do is recompute what the model just said.

Every serving constant is a model constant

The Q&A has a section on how hits are measured that reads like an engineering detail and is actually a statement about how the model was designed. V4.1-Flash is a causal encoder-decoder. Twenty encoder layers compute the global KV cache during prefill. Twenty decoder layers project their global keys and values from the encoder’s final state, so prefill only runs half the stack. Every layer also has a sliding-window attention branch over the last 128 tokens. That window state is the problem: it is per layer, it is not projected from anything, and reconstructing it exactly for a cached prefix would mean running the last 2,560 tokens (20 layers times 128) through the decoder.

DeepInfer does not reconstruct it exactly. It replays only a short suffix, 128 tokens in the Q&A and 140 in the shipped default, and truncates each query’s window to the replayed segment, so the cost of recomputation is “a constant, independent of context length and layer count”. The Q&A says this nearly halves total prefill compute. The cache rules are built around it:

Constant	Value	Where it comes from
Sliding window	128 tokens	model config, `sliding_window`
KV cache block	128 tokens	the storage unit in the cache store
SWA store positions	multiples of 128	stored once after prefill and once at the end of decode
Draft block	5 tokens	DSpark, `DEEPINFER_USE_NEXTN_DECODE`
Speculative backtracking	6 tokens	draft block plus one
Maximum hit length	L minus 134	window plus backtracking
Decoder replay	140 tokens	`DEEPINFER_YOCO_PREFILL_KEEP_TOKENS`, must be at least window plus backtracking
Stored SWA state per sequence	20 layers × 134 tokens × 528 B = 1,415,040 B	the encoder layers only, in MXFP8
Exact reconstruction	2,560 tokens through 20 layers	what the replay avoids

The 128 in the model and the 128 in the storage system are the same number because they were chosen together. So were the 5 and the 6 and the 134 and the 140. This is what co-design looks like from the serving side: the model has a window the cache can store cheaply, and the cache has a block size the window fits in.

Here is what the hit rules do to two ordinary requests. Both continue the same conversation, which so far is 1,460 tokens: a 1,100-token prompt and a 360-token reply. The previous turn left SWA state at two positions, 1,024 after its prefill and 1,408 at the end of its decode.

A · the next turn adds a 300-token tool result · L = 1,760

hit: 1,408 tokens from the cache

300 new

cap L−134

block hit 1,408, SWA state present

B · the next turn adds a 40-token user message · L = 1,500

hit: 1,152 tokens from the cache

308 recomputed

cap L−134

block hit 1,280

no SWA state at 1,280: back off one block

Decoder work, both cases: replay the last 140 tokens only

140 (A)

140 (B)

served from the cachecached but recomputed in the encodernew tokensdecoder replaystored SWA state

The hit rules from the Q&A on two continuations of the same 1,460-token conversation. The hit length is capped at L minus 134, rounded down to a 128-token block, and backed off a further block if no SWA state was stored at that boundary. The 300-token tool result recomputes 352 tokens in the encoder. The 40-token message recomputes 348, because the cap pulls the block hit back to 1,280, where no SWA state was stored, and the rule backs off another block. Grid lines every 128 tokens.

The second case is the one that surprised me. A short follow-up pays almost the same prefill as a long one, because the cap of L minus 134 drops the hit below the last block boundary where SWA state exists, and the rule then backs off another whole block. In exchange the system never stores decoder window state, never has to transfer it, and never runs more than 140 tokens through the decoder at prefill time.

I wanted to know what that trade costs on realistic traffic, so I wrote a simulator that applies the rules literally to synthetic sessions and counts compute in layer-tokens, that is, tokens pushed through one transformer layer. Three workloads: forty-turn agent sessions with tool results of 200 to 4,000 tokens and occasional context compaction, eight-turn chat sessions with short messages, and single fresh documents of 20,000 to 200,000 tokens. Four ways of counting on identical traces. No cache at all. An ordinary 40-layer decoder-only model with a perfect prefix cache, which runs the uncached suffix through every layer. This model with a perfect prefix cache and an exact 2,560-token rebuild of the decoder window. And DeepInfer’s rules. The last three columns are prefill compute as a share of running with no cache at all.

Workload	Hit rate, perfect cache	Hit rate, DeepInfer rules	Ordinary model, perfect cache	This model, exact rebuild	DeepInfer rules
Agent sessions	95.8%	95.7%	4.2%	4.7%	2.3%
Chat sessions	95.9%	90.8%	4.1%	41.1%	7.4%
Fresh documents	22.0%	21.9%	78.0%	40.3%	39.1%

On agent traffic, which is where the revenue is, DeepInfer’s rules cost about half of either baseline, because the decoder only ever sees 140 tokens. On fresh documents they halve the work, which is the “nearly halving” in the Q&A. Short chat turns are the exception. There the fixed per-turn overhead of the cap, the block rounding and the 140-token replay is large next to a 40-token message, and the rules cost 1.8 times what an ordinary model with a perfect cache would. That is still 7 percent of a cold prefill against 4, and a fifth of what an exact rebuild would cost. The policy is tuned for the traffic that pays and accepts a small tax on the rest. The simulator and the traces are in the repo if you want to change the assumptions.

One more thing the replay implies, which the Q&A states and does not dwell on. The decoder’s window state after replay is “an approximate SWA KV cache”. A query at position i within the replayed segment attends to keys from max(s, i minus 127) to i, where s is where the replay started, so the first 127 replayed positions see a truncated window. The served model is not quite the checkpoint. DeepSeek presumably trained for this and the equivalence script in the repo only checks that two implementations of the truncation agree with each other, not that truncation agrees with no truncation.

So I measured it. I rented two B300s and ran that same harness on real text instead of random tokens: twelve prompts of Python, Rust and prose at 256, 1,024, 4,096 and 16,384 tokens, each followed by the 256 tokens that actually came next in the source. The model reads every prompt twice, once with the decoder recomputing the whole prompt, which is the harness’s own exact baseline, and once with the shipped 140-token replay. Each time the harness records the probability the model gave to each true next token.

Read the numbers as a ceiling on what the shortcut costs, not as its exact cost. The shortcut runs through different code from the exact path, and different code rounds differently, so part of every gap below may be rounding rather than truncation.

Over 3,072 positions	140-token replay against exact
Perplexity on the true continuation	0.36% higher (95% interval 0.0 to 0.8%)
Top token changes	1.1% of positions
Probability of the true token moves by more than 10%	10.5% of positions
Bit-identical to the exact run	6% of positions
Cost at 16,384 tokens against 256	about the same

The truncation is close to free on quality, which is presumably why it ships switched on and the Q&A can treat it as a detail. It is not free on identity, though this test cannot say how much of that is truncation and how much is rounding. At a 1 percent chance per position of a different top token, a greedy 256-token answer from the shipped path and one from the exact path agree all the way through about one time in twenty. I did not test cache hits; every prompt here was fresh. On a hit the decoder still replays the same 140 tokens, but the encoder resumes from a sliding-window snapshot saved by an earlier request, so whether a hit and a cold prefill agree exactly is an open question. Either way, if you compare DeepInfer’s output with the same checkpoint served some other way, expect the two to drift apart in most long answers, and do not read that as a bug.

The caveats are the reason for the ceiling. There was one seed and twelve prompts. And there was no control run: the same harness with 512 tokens of replay on the 256-token prompts, where the truncation is mathematically a no-op, would have measured the rounding alone. I cut it to keep the rental short. Taking that noise out should only lower the flip rate and raise the bit-identical share. The perplexity change is an average of gains and losses, so noise moves it either way, and its interval already reaches zero. The per-prompt ratios run from 0.994 to 1.024, which is what a small effect, or none, plus noise looks like.

Getting it to run at all was a finding of its own. DeepInfer reads the model’s Engram tables, two hashed n-gram lookup tables of 183 GiB between them, through DeepEP over RDMA, and that path will not start without an InfiniBand device, even when both GPUs sit in the same box. The engine assumes DeepSeek’s network. On a cloud node without one, I replaced the Engram fetch with a local gather that returns the same tensors and left everything else alone. The patch and the harness are in the repo.

Determinism costs you speculation

Two of the features in the README are easy to pass over and worth a paragraph each.

Batch-invariant inference, enabled with one environment variable, makes a request’s output independent of what else was in the batch. Thinking Machines wrote the canonical post on this in September 2025, showing that most “nondeterminism” at temperature zero is really kernels changing their reduction order with batch size, and that fixing it cost them about 1.6 times in end-to-end throughput on a demo. DeepInfer ships it as a production flag, and its evaluation harness has a --batch-invariant option labelled “slower”. The README marks it NVIDIA-only, and the repo’s own tests and equivalence scripts switch it on before they compare outputs.

Deterministic sampling needs the same flag plus a fixed seed per sequence, and it needs DEEPINFER_ENABLE_MTP_VARLEN=0, which switches off variable-length verification for DSpark. So you can have reproducible outputs or confidence-scheduled speculation, not both. That is a real trade. The thing that makes speculation efficient on a shared fleet, deciding per step how much of each draft to verify based on load, is also a source of batch-dependent behaviour.

On DSpark itself I will not repeat what I wrote in June, except to note that the Q&A now contains DeepSeek’s own production table for the claim I made there, that speculation is a scheduling problem. Dynamic-length verification cuts the draft tokens verified per step from 5.0 to between 3.2 and 4.4 and leaves the accepted length almost unchanged, 3.2 against 3.3. DeepSeek’s estimate of the throughput change is minus 1.5 percent at 16 sequences per rank, plus 5.4 percent at 60, and plus 11 percent at 96. Below the point where the GPU is busy, scheduling speculation loses a little. Above it, it wins, and it wins more the busier the fleet is. There is also a detail I liked: because scheduling is asynchronous, the confidence scores for step N have not arrived when the CPU schedules step N plus 1, so the CPU only fixes the total verification budget and a small kernel on the GPU recomputes the per-request lengths with a cumulative product, a global sort and a scatter.

What to do with this

If you are deciding whether to run DeepInfer, the Q&A answers that for most readers in one line: the smallest efficient deployment is 64 prefill GPUs plus 64 decode GPUs, plus the storage, the balancers and the autoscaler you write yourself. Below that scale you are better served by the engines that already run the model on more hardware, and the release is explicit that it was “originally built primarily on vLLM” in any case.

If you are building a serving system of your own, the release is worth a weekend regardless of whether you ever execute it:

Read the Q&A before the code. It is the design document for the half of the system that matters, and it is unusually willing to give numbers: moe_lb of 1.05 to 1.09 for expert balancing, 400 Gbps per 8 GPUs for cross-datacenter prefill and decode, 14.4 TB of logical cache per GPU, a one-minute grace period that forwards unfinished requests when a decode engine is told to shut down.
Measure your own hit rate against your own prices. The instrument above is a model of DeepSeek’s business, but the shape of the argument is general. If your traffic is agentic, the prefix cache is not an optimization, it is the cost structure, and it belongs on a storage cluster that every engine can reach.
Pick your cache granularity and your model’s window together. The 128 story is the part of this release I expect to be copied most, and it only works if the two teams agree on a number before the model is trained.
Treat the request edge as part of the contract. The companion deepseek-recipe library renders the prompt and parses the output exactly as DeepSeek’s API does, down to which bytes are allowed to become control tokens. I have a probe in the repo that shows what goes wrong when you tokenize its rendered prompt the obvious way, and it deserves a post of its own.

DeepSeek has said the inference code for future models will land in the same repository. The stubs in the engine are shaped like the system the Q&A describes. I would not be surprised to see them filled in, and I would not expect the cache to follow.

Sources

DeepInfer repository: github.com/deepseek-ai/DeepInfer, and its Inference Deployment Q&A, which is the source for every production number quoted here
deepseek-recipe: github.com/deepseek-ai/deepseek-recipe
DeepSeek’s architecture write-up (Chinese): DeepSeek V4.1 Flash 大规模推理部署实践
DeepSeek API pricing, read 24 October 2026: api-docs.deepseek.com/quick_start/pricing
DeepSeek-V4.1-Flash technical report: arXiv 2609.19969
DeepSeek’s February 2025 day of traffic for V3 and R1: DeepSeek-V3/R1 Inference System Overview
The checkpoint and its config: DeepSeek-V4.1-Flash-1017 on Hugging Face
DualPath, the decode load balancer the Q&A points to: arXiv 2602.21548
Thinking Machines, “Defeating Nondeterminism in LLM Inference”, 10 September 2025: thinkingmachines.ai
My June post on DSpark: Speculation is a scheduling problem
The economics model, the cache policy simulator, the recipe probe and the replay harness: github.com/bassrehab/deepinfer-lab

Cite this article

Mitra, Subhadip. (2026, October). DeepInfer: The Engine Is Open. The Cache Is the Business.. Subhadip Mitra. Retrieved from https://subhadipmitra.com/blog/2026/deepinfer-cache-is-the-business/

@article{mitra2026deepinfer-the-engine-is-open-the-cache-is-the-business,
  title   = {DeepInfer: The Engine Is Open. The Cache Is the Business.},
  author  = {Mitra, Subhadip},
  journal = {Subhadip Mitra},
  year    = {2026},
  month   = {Oct},
  url     = {https://subhadipmitra.com/blog/2026/deepinfer-cache-is-the-business/}
}

What shipped, and what was described

The two numbers

155 trillion tokens a day at $38 a GPU-hour. What mix of tokens makes that true?

Then and now

What a cache hit actually is

Every serving constant is a model constant

Determinism costs you speculation

What to do with this

Sources

Cite this article

Get More Like This

Continue Reading

DeepSeek DSpark: Speculation Is a Scheduling Problem

What Runtime Interpretability Actually Costs, Part 1: The Case for Measuring It

The Stateless MCP Spec Audited My Compiler, and My Confirmation Gate Failed