Non-deterministic Qwen3.8-Flash-Next: we found the faulty vLLM kernel and fixed it

Under greedy decoding (temperature=0) the same prompt should produce the same output. Ours didn’t: Qwen3.8-Flash-Next (the RadixArk NVFP4 checkpoint) on vLLM, on a GB10-based DGX Spark, gave a different output on different runs for 13 out of 50 tasks, and on five of them the extracted date or amount changed too. This post is about how we narrowed the bug down to a single CUDA kernel, what the fix cost, and what else surfaced along the way that had previously looked like noise.

In short:

• It’s not the measurement and not the model: three control engines, including the very same checkpoint on llama.cpp, at 0/50. The bug lives in the serving stack.
• The root cause is the QSA sparse-attention indexer’s persistent_topk CUDA kernel: under a race condition it picks a different top-2048 position set on every run, so even the prefill logits are non-deterministic.
• The fix: replace the selection with torch.topk + canonical sorting (a single-file overlay, VLLM_QSA_EXACT_TOPK=3). Bit-exact deterministic; prefill goes from 2,360 to 1,760 tok/s (1.35×); MTP and cudagraph stay. On the main suite, 98.00/100 (the attainable maximum), 0/50 unstable; that’s actually one point better than the faulty kernel, where majority voting crowned a wrong date the winner.
• Bonus finding: the deterministic prefill exposed a second failure mode that had been masquerading as randomness (a greedy + thinking repetition loop), and it turned out that MTP speculative decoding is not output-equivalent to greedy on this model.
• Quick test for your own stack: the same prompt ten times, temperature=0, max_tokens=1, top_logprobs=20, one request at a time. If the top-20 logprob list isn’t bit-identical, your prefill is non-deterministic.

Setup

• Model: RadixArk/Qwen3.8-Flash-Next-NVFP4; a 177 B-parameter MoE, of which ~51 billion is an n-gram table (PLE). That doesn’t fit in memory, so it’s served from disk via mmap.
• vLLM 0.1.dev20073+g8e685d198 + the mmap-PLE patch from the blazux/qwen3.8-Flash-DGX recipe (82ed48d), which doesn’t touch the QSA code path
• Hardware: GB10 / DGX Spark, driver 580.173.02, 121 GB unified memory
• Launch: --max-model-len 262144, --max-num-seqs 2, --no-enable-prefix-caching, chunked prefill 8192, cudagraph PIECEWISE, MTP=2 speculative decoding, NVFP4 MoE backend FLASHINFER_CUTLASS
• Measurement: Hungarian document extraction (KIE) eval, 65 tasks, all greedy, the main 50 items run three times

How we noticed

The harness sends every item three times, scores each run separately, and stores the SHA of every output. That’s where something showed up that shouldn’t exist under greedy decoding: 13/50 items with a different output on different runs. On five of the 13 the extracted value changed as well: a deadline of 2026-11-10 in one run and 2026-11-05 in another; a grand total of 6,300,000, then 5,940,000. On one item two of the three runs were wrong, so majority voting would have crowned the wrong answer. Two runs also drifted off into 16,384 tokens of empty thinking (finish_reason=length, empty content); that later turned out to be a failure mode of its own, not a symptom of the kernel bug.

The same item set, the same harness, on three control engines:

Making sure it wasn’t me

Before blaming the stack: the payload is an explicit temperature: 0.0, top_p: 1 on every engine; the prompts are static and the request is built once, before the run loop; the vLLM log shows Running: 1 reqs throughout, so batch composition couldn’t have changed. The length hypothesis (“longer generations are less stable”) was refuted by the data: unstable and stable items have practically the same prompt length (medians of 6,038 and 6,072 tokens), and a 10,178-token generation repeated bit-identically three times.

Where it diverges

The harness also stores the reasoning content (reasoning_content), so we can see where the output splits: on 12 of the 13 items, at the very first tokens of the thinking, not as accumulating drift. The model starts thinking in a different language on different runs, for the same Hungarian question:

run 0:  A felhasználó azt kérdezi, hol találom meg azt a szabályt…
run 1:  We need answer user's question in Hungarian, based only on…

Under greedy decoding the first token is the output of the prefill. With a deterministic prefill, even a tie is always broken the same way; here it wasn’t, so the prefill logits change from run to run, and decoding merely inherits that.

One switch at a time

Isolation on the 13 unstable items, five runs each, exactly one change per round:

As a side finding, the empty 16,384-token runaway only ever happened under MTP (0/65 runs without it); what that actually means became clear during validation (see the second failure mode below).

The probe that would have shown it in two minutes

A far more sensitive and cheaper instrument than the suite is a prefill probe: the same prompt, temperature=0, max_tokens=1, logprobs=true, top_logprobs=20, ten times, one request at a time, then a bit-level comparison of the top-20 (token, logprob) lists. We ran it on six prompts, with the stock and the replacement kernel:

The key finding: we deliberately included two items in the probe that had looked perfectly stable throughout the suite, and their prefill also produced ten different logit vectors in ten runs. The non-determinism was universal; the suite only revealed it where two tokens were tied. That’s why every new serving recipe should be measured with the probe first and the suite second: two minutes, independent of batching effects, measuring prefill determinism directly.

What persistent_topk is, and why the Spark in particular

Flash-Next handles long context with sparse attention (QSA): for every query, an indexer selects the 2048 positions to attend to. The code path in vLLM is qsa.py::qsa_select_paged_tokens, and on GB10 (compute capability family 12x) the use_cooperative_topk branch is false, so the torch.ops._C.persistent_topk kernel runs, which assigns slots with atomic operations (42 atomicAdd calls in the kernel source).

An intermediate experiment ruled out that the bug is merely slot ordering: taking the stock kernel’s output with the set untouched and re-sorting it canonically afterwards, 3 of 6 prompts remained unstable. Under a race, the kernel changes the selected set, not just its order: attention works from a different context on every run. This is consistent with the correctness issue already open against the kernel (vLLM #51782: the histogram-based selection silently drops genuine top-k candidates when values fall into a shared bin); a race at the bin boundary keeps different candidates on different runs, and the longest prompt, at 24k tokens, was the worst. The size of the logit discrepancy (0.25–3 nats) confirms it too: this isn’t floating-point rounding noise, it’s a different attention context.

The price of the fix

The replacement is a single-file overlay on the installed qsa.py, behind an env switch: VLLM_QSA_EXACT_TOPK=3 = torch.topk(sorted=False) radix selection, followed by two stable sorts (ascending by index, descending by value) and -1 padding when length < k. In unit tests and on the probe it’s bit-identical to the full-sort reference.

The swap only affects the prefill-side indexer: MTP and the PIECEWISE cudagraph stay, and the suite’s output speed moved from 26.4 to 24.9 tok/s (the latency-based metric includes the slower prefill). The 13 unstable items go to 0/13 with the fix, five runs bit-identical. The full re-measurement on every suite:

The main suite’s 98.00 is the attainable maximum (for technical reasons, the scorer’s two LLM-judge points are unavailable to every engine), and the score went up with the fix: under the stock kernel, the majority voted for the wrong date on item T3-05, while the deterministic run gets it right. And behind the hard suite’s 90 points is not a regression from the top-k swap but a second failure mode that determinism itself made visible; that’s the next section.

The remaining 26% prefill cost is the price of non-native selection. Our upstream ask to vLLM: a deterministic (index-ordered tie-break) path in persistent_topk, the way FlashInfer did it for its own sparse-attention top-k (flashinfer #2661), or a documented switch for exact selection. We filed the bug, with the reproduction and the fix, in the recipe’s repository: blazux/qwen3.8-Flash-DGX #3.

A by-product of determinism: a second, previously hidden failure mode

The hard suite’s loop item failed identically in all three runs: 16,384 tokens of thinking, an empty answer, bit-identical content. The end of the thinking is the same ~180-character paragraph repeated 22 times: the model spins on the format of one field in the output schema and never closes its thinking. This isn’t the top-k kernel’s fault; it’s the known bad pairing of greedy decoding and thinking mode (Qwen’s documentation explicitly advises against greedy in thinking mode).

The instructive part: this failure mode existed under the faulty kernel too, it just looked random: it surfaced once in 150 runs, and on a different item each time in the language-comprehension set (one task with the stock kernel, another with the fixed one). The noise of the non-deterministic kernel would occasionally knock the model out of the loop. So the deterministic prefill didn’t cause this bug, it made it reproducible: what used to be a sporadic, ungraspable phenomenon now shows up identically on 3/3 runs, which makes it detectable (repetition ratio in the thinking) and manageable (retry with T>0, presence_penalty, or disambiguating the prompt).

The item-level follow-up test taught a second configuration lesson as well: with the fixed kernel and MTP off, the loop disappears (10/10 runs, 864 tokens, correct JSON), and the MTP=2 thinking departs from the MTP=0 trajectory around the fifth token. In other words, speculative decoding on this model and build is not output-equivalent to pure greedy decoding; it was the diverted trajectory that led the model into the loop. The suite-level MTP comparison is still running, so we won’t claim more than that.

The accurate summary, then, isn’t that the kernel swap solved every problem, but this: the non-determinism is gone, and with it two configuration problems that had been disguised as noise (the greedy + thinking loop tendency and the MTP divergence) became measurable and manageable.

Side thread: so which model is better?

The measurement started life as a four-way comparison in the 128 GB, run-on-your-own-hardware class, on 65 Hungarian tasks: Flash-Next NVFP4 297/300, Qwen3.6-35B FP8 (our current production model) 294, Flash-Next IQ4_XS 289.33, Qwen3.5-122B 275. The ranking effectively comes down to four items; that’s how tight the field is. Two operational facts behind the scores: IQ4_XS is dragged down by quantization damage on a single hard item (3.33/10 where NVFP4 is flawless), while the 217-thousand-token document bundle takes the vLLM stack 157 s and llama.cpp 1,121 s. The detailed comparison will be a separate post; the point here is only that the best-scoring configuration was the only non-deterministic one, and you can’t see that from a leaderboard.

In summary

A greedy run that gives a different answer every time isn’t “LLM uncertainty”; it’s a bug in the stack, and it can be tracked down. In our case it was a sparse-attention top-k kernel, and the fix was one file and one environment variable. What carries over beyond this particular model:

1. Under greedy, divergent output is always a bug signal, never natural variance. If the control engines give 0/50, the bug is in the server, not in the model and not in the measurement.
2. The prefill probe is cheaper and more sensitive than the suite. Ten requests, one token, top-20 logprobs: in two minutes it shows what the suite only reveals on tied items. Run it first on any new serving recipe.
3. A leaderboard doesn’t show determinism. Of the four candidates, the best-scoring one was the only unstable one; a score needs a stability measurement next to it.
4. Determinism doesn’t just fix, it exposes. What looks like a sporadic failure in a noisy system (a loop, an MTP divergence) is reproducible in a deterministic one, and therefore manageable.

---

Measured on a GB10 / DGX Spark (driver 580.173.02, 121 GB unified memory), under vLLM 0.1.dev20073+g8e685d198, with the RadixArk/Qwen3.8-Flash-Next-NVFP4 checkpoint, MTP=2 speculative decoding and PIECEWISE cudagraph mode. The corpus is 65 Hungarian document-extraction tasks, all greedy; the main 50 items ran three times, the isolation ran five times on the 13 unstable items, and the prefill probe ten times on six prompts. The control engines received the same item set through the same harness. The suite-level MTP comparison was still running at the time of writing, so the claim about MTP rests on an item-level follow-up test.