The prefix cache changed the answer — but only when another request had warmed it

Two weeks ago I wrote about our Qwen3.8-Flash-Next deployment on a DGX Spark returning different dates from the same Hungarian invoice at temperature=0, and how we traced it to the persistent_topk kernel in vLLM's sparse-attention indexer. That fix has been in production since; the deterministic kernel itself is going upstream as an open pull request (#55122), to which we attached our measurements as a comment on 3 September.

This post is about what was left after the fix.

In short:

• In the quality suite, all fifty answers have been stable ever since. In the logit-level probe, one of the four items tested has not: the same one, across two measurement rounds and every fresh server start, always with the same shape.
• Since the previous round I had assumed the “cold” path was the suspect. It is exactly the other way round: the cold and partial-hit paths give the correct result, and the full cache hit diverges — when the shared prefix blocks were written by a request of a different length.
• Turning prefix caching off removes it. This is not the well-known, long-fixed “all-zero state” bug: that fix is present in the system under test.
• On an entirely different stack — official release, different engine version, deterministic kernel — the same task fails in the same way.
• We could produce the bug on demand on a document never used before, purely by sizing the requests. On a third document it did not appear — so the explanation is incomplete, and I write that up in the same detail.
• Bonus: the deterministic kernel processes input 21–28% faster than the safety fallback we run today, at unchanged determinism.

Update, 14 September 2026: we have the mechanism, and a single line removes it. Below, this article describes a prediction that failed on a third document, and concludes that the explanation is therefore incomplete. It is complete now: the variable is not the difference in block count, but whether the first request checkpoints its state at the shared block boundary. Details and the measurement of the fix: Update, 14 September.

Why “the answer is the same” is not enough

My first instinct was to look at the text. The text was identical across all ten runs. Stop there and you write “all good” in the log.

So the probe does not look at the text. For every generated token it pulls the top twenty candidates with their log probabilities, and hashes the whole sequence with SHA-256. If the model picks the same word but the runner-up has moved, this sees it. A string comparison does not.

That distinction is not academic. Under greedy decoding the argmax can stay stable while the logits drift — and then flip on the next token. In the very first round, before the top-k kernel fix, a logit-level drift of the same kind moved the extracted date or amount on five of fifty items.

Text equality is not evidence. That is rule number one in our measurements now.

The experiment that settled it

The stubborn task (call it B) extracts fields from a Hungarian document. Another task in the set (A) is handed the same document, with a different question and a different response schema. In the measurement order, A ran first.

That looked suspicious: if A had already written the shared document's blocks into the cache, then B's “first, cold” run was never cold at all. I ran three arms, each on its own fresh server start:

Read the middle row again. The first run of the contaminated arm — the one that gets a partial hit on another request's blocks and prefills its own continuation fresh — reproduces the fully cache-free clean arm bit for bit. The second and later runs, the full cache hits, are the ones that diverge, starting at token zero.

So the fault line is not “cold versus cached”. In the clean arm, runs 2–10 are also full cache hits, and nothing goes wrong there. The difference is whose request wrote the shared prefix blocks. Same request: the read-back is correct. A different request of a different length: the restored state produces different logits.

The third row closes the chain: with prefix caching off, the same sequence is ten-for-ten identical.

Why a hybrid model specifically

Qwen3.8-Flash-Next is not a plain transformer: linear (Mamba-style) layers alternate with sparse attention. Caching is harder for such a model than a plain KV cache, because the linear layers carry state that has to be saved at block boundaries and restored later.

vLLM handles this with a dedicated mode, and — this is the part that matters — it turns it on by itself as soon as prefix caching is enabled:

INFO [config.py:605] Mamba cache mode is set to 'align' ... when prefix caching is enabled

In the control run, with caching off, that line never appears. Which is why the two configurations' absolute hashes cannot be compared: a different numeric path is running. The metric is stability, not the value.

And here comes the least comfortable turn in the story. Two open vLLM pull requests target exactly this area: one (#54076) ties align-mode chunk splitting to the cache group's block size, the other (#53798) seeds the linear layer's state index correctly. I went through the recipe our deployment is built from (blazux/qwen3.8-Flash-DGX) — and it turns out the substance of both fixes is already in the system I measured, as a two-line patch, in exactly those two files.

The patch's own header says what it cures: the engine overwrote the block size with the smallest cache group's size, while saving and restoring the state assumed the large one — the result was an all-zero restored state on every cache hit. A spectacular, easy-to-spot failure. Fixed long ago.

What I am measuring sits beyond that: the all-zero case is handled, and a subtler difference remains, visible only across requests. This does not make the two open fixes pointless — they do considerably more than the two-line patch, and may well cover the remainder. It only means the obvious explanation (“surely it's that old bug”) does not hold here.

The prediction that held — and the one that did not

A bug is properly demonstrated when you can produce it where it was not before.

The log gave away the cache block size: 1600 tokens. I then looked at the task pairs I had measured. The failing pair was the only one where the two requests seal a different number of full blocks:

Hence the prediction: take a document that has never been used on this server, size the two requests' tails so they seal a different number of full blocks, and the divergence should appear.

I sized them, ran it — and it did appear, with exactly the same shape. Same document, equal block counts: stable. So far the prediction holds.

Then I tried a third document, on the same server start: 2 blocks vs 3 blocks — and it stayed stable. Synthetic, machine-generated filler text did not trigger it either.

So crossing a block boundary looks necessary but not sufficient: the document's length or content matters too. The most likely additional factor is the sparse-attention block selection, which itself depends on the context — separating that out is the next round's job.

I am writing this mixed result up in as much detail as the successful prediction. A hypothesis you have not tried to break is not a hypothesis; it is a hope.

Two days later it turned out the hypothesis was not half-right but badly worded. The correct rule explains all three cases, including the third document that appeared to break it: see the 14 September update.

The control that mattered most: an entirely different stack

A measurement is worth something when you can show the bug outside your own build. So I brought up a second server — same machine, same model files, different in everything else:

The result: the same task fails, with the same shape. And the predicted case — the request pair sized onto a block boundary — appears here too, while the equal-block-count control stays stable on that very same server.

So the effect is not a quirk of our build, not of the older package version, and not a side effect of the slower top-k workaround.

A side result that pays for itself

Because the second stack uses the deterministic kernel (the kernel from #55122, packaged in jschmied/qwen38-flash-next-gb10), the same probe could measure what determinism costs us today. Same machine, same model, same flags, six input lengths, all cold:

Memory stays on a flat plateau on both stacks, with no step per context length; no preemption and no crash in any of the seven container logs. Determinism no longer costs anything — but our production stack is still paying for it until we switch. (The condition for switching is re-running the quality suite: the two exact selections break ties differently, so the score does not carry over.)

What to take from the method

Your own instrument can contaminate the measurement. The stubborn task “failed” because another task in the set had run before it on the same document. The model was not unstable — my measurement order had created a state I had not accounted for. The remedy is simple: each probe uses a document nothing has touched on that server start, and each arm runs on its own fresh start.

Not all cache hits are alike. “Same prompt, therefore same answer” breaks on a hybrid model when another request has warmed the shared prefix. If you run several tasks against the same document — and a document-processing system does exactly that — this is not a theoretical risk.

Version archaeology is measurement too. Along the way it turned out the model's package inside the engine had been renamed. That has two practical consequences: one open fix (#56500, the bounded sparse-attention prefill workspace) cannot be layered onto the release (the file it would modify does not exist there), and with the old launch parameters the new release fails silently during graph capture. Neither shows up in a changelog; you have to look inside the running system. We posted that negative result and the compatibility limit as a comment on the pull request, so the next GB10 tester does not walk into the same wall.

Update, 14 September: we have the mechanism

After this article went out, an external review pointed at a specific place in the engine's source worth looking at. I looked, and the claim held up: the mechanism is in the request scheduler, not in the sparse-attention block selection this article guesses at above.

In a hybrid model the cache holds two different things about a shared prefix: the attention key/value blocks, and the recurrent layer's internal state at block boundaries. The two are not produced together. The scheduler processes a prefix in chunks, and can only save the state where a chunk ends exactly on a block boundary. With speculative decoding, however, the scheduler backs off by one full block from the last cacheable position, because the attention side still needs that last block for the draft.

The consequence is a sharp threshold. Our block size is 1600 tokens, so:

• a request below 3200 tokens runs as a single chunk and saves no state at the 1600 boundary, even though it does write its key/value blocks there;
• a request above 3200 tokens stops at 1600 and saves.

So if request A is below the threshold and B is above it, then B itself produces and stores the state at 1600, while on its own repeat it gets the key/value blocks back from A's earlier copy. Two tensors of different origin meet inside one restored prefix.

Backwards first: the rule explains the case that broke the old one

I extracted the function from the source of the actually running server and ran it, without a GPU, over all eight request pairs from the earlier measurements. The question for each pair: did request A checkpoint its state at the shared boundary?

Eight out of eight, including the pair that broke the prediction in this article: it stayed stable because A (4,689 tokens) is well above the threshold, so it did write the state at 1600, and B reads back that same copy. The difference in block count was only a symptom of the real rule.

That is the useful lesson about hypotheses: my old rule was not “nearly right”, it measured the wrong variable. Both rules fit the same seven data points; the eighth let only one of them through.

Then forwards: nine tokens flip it

If the threshold is real, then a handful of tokens at the end of request A must flip the outcome, with the shared document and request B unchanged. Measured on a fresh server, each cell with its own cache salt so they cannot contaminate one another:

The first two rows are the point: same B, same shared document, request A differing by nine tokens, and that decides whether the system is deterministic.

A number I had not measured: the actual hit

My earlier probes worked from the completion and never asked the server how many tokens it actually served from cache. The external review was right to flag that, so I enabled the corresponding reporting. In the failing case:

This also corrects a claim made above. The article calls B's first run a “partial hit”. The hit is in fact zero: the literally identical first 1600 tokens written by A were in the cache, but the hybrid lookup will not use them without the matching state. That is why B's first run is bit-identical to the fully cache-free arm: because it really is cold. The old measurement was sound; my reading of it was not.

The logits really do move

The other fair objection: the probe hashes the twenty candidates in the server's own order, so a mere reshuffle of tied candidates could in principle produce a different digest. So I stored the raw lists too. For the first three generated tokens in the failing case:

Every shared candidate gets a different value, the candidate set itself changes, and the digest taken over token-id-sorted lists also has two variants. So this is a genuine numeric shift, not list ordering. The chosen token happened to be stable here, which is why nothing showed in the visible text.

The fix: one line

The engine has an open fix, #54076, which among other things rewrites exactly this condition: every crossed block boundary must end a chunk. The pull request's own rationale states the mechanism, that a chunk spanning several blocks leaves the interior state slots permanently null. This fix is missing from the two-line patch in our recipe.

I started a second server from the same build with the same flags, changing that single condition. The result:

The last row matters most. That is the same item pair, with the same probe, that failed on two different stacks across three measurement rounds. On the patched server, ten runs out of ten are bit-identical.

What this still does not tell me

The root cause is in the scheduler, but I have not measured which operation produces the first differing number under different chunking. It is telling that on the patched server the cache-hit counts stay the same, 0 then 1600, so block origin is still mixed while the result is identical. The most likely reading: mixed origin is not the problem by itself. In the original case the shared block's key/value data was written by a single 3,196-token chunk while the state came from a 1600-token chunk, and the same tokens under a different chunk shape give different numbers. That is an inference from behaviour, not tensor-level proof.

One important limit: for us this is an experimental backport, one line out of an open pull request, not the full fix. It is not what we put into production; we wait for the fix to land in the engine the normal way. We have also not yet measured the cost of stopping at every boundary (more scheduling steps).

What we do with it

Prefix caching stays on in our production deployment: the measured effect is in the logits, and the safeguards were already in place — three runs with an agreement check for extraction, a separate verification step for chat. Turning it off would cost an order of magnitude in latency on the long, repeated prefixes that dominate our traffic.

The measurement continues. The 14 September round has already shown which open fix matters to us, and that rewriting a single condition removes the phenomenon. The next step is the same measurement on a runtime carrying #54076 in its full form, together with a re-run of throughput and the quality score.

And we give all of it back. All three results went upstream as comments on the corresponding vLLM pull requests, with raw data and the exact launch commands: the cross-request reproduction on #54076 (referencing #53798 and this article), the GB10 negative validation and compatibility limit on #56500, and the deterministic kernel's +21–28% prefill gain on #55122. A local LLM stack becomes trustworthy when the measurements are not kept on our own blog but taken to where the code is written.

---

Measured on a GB10 / DGX Spark (driver 580.173.02, 121 GB unified memory) with the RadixArk/Qwen3.8-Flash-Next-NVFP4 checkpoint, on two separately built engines (vLLM 0.1.dev20073+g8e685d198 and the official v0.29.0), MTP=2 speculative decoding, PIECEWISE cudagraph mode, across seven fresh server starts. The probes hash the per-token top-20 logprob list of the whole completion; identical visible text is never treated as evidence. The 14 September update was measured on two further fresh server starts, with the same model and flags, and with the server's cache-hit reporting enabled. The mechanism still has no kernel-level instrumentation (which operation yields the first differing number), and the quality suite was not re-run on the patched arm.