Honest, detailed technical articles from the development of DocAI: LLM inference tuning, GPU optimisation, document processing pipelines, enterprise AI architecture. Negative results and lessons learned included. New post every week.
We fixed the non-deterministic top-k kernel, and one task out of fifty still wobbled. Since the previous round we had assumed the cold path was the suspect: it is exactly the other way round. The failing path is the full cache hit, when the shared prefix was written by a request of a different length; turning prefix caching off removes it, and the fix for the known “all-zero state” bug is already in the system. We reproduced the effect on an entirely different stack, and produced it on demand by sizing two requests onto either side of a 1600-token block boundary. Update (14 Sep): we have the root cause, a checkpoint the request scheduler skips, and a single line removes it: the original failing item is 10/10 bit-identical on the patched server. Bonus: the deterministic kernel is 21–28% faster than our current fallback.
Temperature 0, identical prompt, and the model answers differently on every run: 13 of 50 tasks unstable, with the extracted date or amount changing on five of them. The same checkpoint on llama.cpp: 0/50, so the bug lives in the serving stack. Flipping one switch at a time led us to the QSA sparse-attention indexer’s persistent_topk kernel, which picks a different context on every run under a race condition. The fix is one file and one env variable at a 1.35× prefill cost: 0/50 unstable, and a score one point higher. Along the way, determinism exposed a second failure mode that had been masquerading as randomness, and gave us a two-minute probe that works on any stack.
“The tales of A nagy ho-ho-ho-horgász were written by Attila József” — that is what a current AI model answered in Hungarian, with complete confidence. In English and Chinese it named Radnóti, and the chat-tuned variant invented an author complete with dates. Nowhere an “I don’t know”. We measured 70 questions in three languages to see what the “prompt in English” advice is worth — and looked inside all 32 layers of the model. We found no sign of an English waypoint; what sits in the middle is not a translation but a shared working session. Facts documented only in Hungarian, however, are what the model knows least well in its own language: seven out of a hundred.
When a model doesn’t do what you want, you have two levers: train it, or filter its output. We measured both on Hungarian verse, where “good” is decidable with a ruler. What is measurable, the filter buys more cheaply than 64 minutes of GPU time — and with a third as many invented words. Voice, though, cannot be bought: there the filter does not move at all (56.4% → 56.0%), and recognisability goes from 56% to 92% purely from training. Plus the textbook metric that picked the wrong model.
We put the twice-as-large Llama-3.3-70B up against our production Qwen3.6-35B-A3B on 100 real Hungarian documents. The expected result: 30 F1 points behind and a 26× slowdown. The unexpected one: the single dimension where Llama looked better was the output of an error detector — and once we checked ground truth, that number turned out to show the exact opposite of reality. Plus the formula that tells you in five minutes whether a model is even in the game on your own hardware.
A vendor publishes an “agent-ready” serving recipe for our model family. We measured all three flags on our own FP8 model: −2.2%, −2.4%, −3.6% — none of them made it in. The interesting part is the third arm: a longer speculative draft is +19% on JSON extraction and −9% on short chat, because the fault line runs along acceptance rate. Plus an uncomfortable bit of self-examination: we measured at concurrency 8 while production peaks at 3.
The same model in NVFP4: +40% throughput, +69% single-stream extraction — and block-FP8 stayed anyway. Our established F1 measurement couldn’t decide (after normalization the gap is 0.0001), and neither could our conversation eval. A third measurement made the call: on 100 real documents, the “which company is us” error jumped from 7 to 12–18. Plus a methodological reckoning and one piece of good news: the error is total, so three lines of code catch 100% of it.
Two vendors, the same DGX Spark, opposite launch commands for the same model — one forbids exactly what the other prescribes. We measured it: the wrong backend is a 2.0–3.3× slowdown, 10 measurement points, 0 crossovers. Then the twist: the same marlin backend wins on NVFP4 and loses on block-FP8. The right backend isn’t a property of the hardware, but of the hardware and the quantization format together — and no vendor model card says so.
Magyar Telekom says token cost is the biggest AI risk — nobody knows how its price will move. We looked at our own numbers: two DGX Sparks, 30 days of Prometheus data, GPU averaging 11.56 W, the whole system ~100 kWh/month, i.e. roughly 6,500 forint. I am not claiming it is cheaper — only that it does not move. The box has a ceiling, the token bill does not. Plus the engineering honesty: when NOT to do it this way.
In 2026 the question is no longer whether to use AI, but where the data ends up. We walk through what is going live now under GDPR and the EU AI Act after the Digital Omnibus, the five real pain points (legal basis, information, data-subject rights, automated decisions, third-country transfer), and why most of compliance is not a legal but an architectural question. Plus: a practical checklist and the decisive question — cloud or on-premise?
This month’s DocAI developments: Partner 360 brings every detail about a partner onto a single screen, company-data lookup now includes EU VIES checks, uploaded bank statements are matched to invoices automatically, there are new contract and payslip registers, and the chat assistant now answers questions about specific invoice line items. Plus: documents outside the NAV system and a Telegram integration.
What is AI document processing, and how do OCR, IDP and KIE differ? We walk through the market’s three solution types — the cloud hyperscaler (Azure Document Intelligence, Copilot), the consultancy project and SharePoint + RAG chat — and where all three give way: the data leaves the company, there’s no Hungarian-language optimisation, the solution is never truly yours. Plus DocAI’s measured answer: on-premise, F1 = 0.975 on a Hungarian corpus.
Eight partner companies, one day from the familiar DocIT foundations to a live DocAI demo. A computer science class from LLM to agent, spontaneous questions run live, a DGX Spark that drew attention even switched off — and an idea that grew out of the conversation: a NIS2 audit module we’re starting this summer.
A systematic head-to-head between a 35-billion and a 122-billion parameter model on DocAI’s own Hungarian business eval harness. The larger model didn’t win on structured extraction and ran 2.6× slower — but on complex multi-step analyses it gave a better answer on five out of six tasks, with five times fewer data lookups. Both models in production, for different purposes. Plus the pitfalls: NVFP4 on sm_121a, FlashInfer MoE, KV-cache FP8 trade-offs.
How did Gree Hungary in Szarvas turn a multi-day approval process into same-day flow? 30–70 documents per day, automated partner verification against four authoritative public sources, 1+ hour daily savings on payments alone, 3–4 month payback — in the honest words of our first customer.
Gemma4 candidate eval on the DocAI Hungarian invoice KIE corpus: F1 0.890 vs Qwen3.6 0.975, single-stream decode 30-80% slower. But the speed measurement’s side effect revealed that MTP acceptance rate on the JSON-KIE workload is 99% — the previous article’s 72.5% global figure hid this completely. The DocAI workload is MTP’s architectural best case.
Qwen3.5-122B-A10B NVFP4 on a single Spark, vLLM 0.19.2, with MTP: 30 tok/s JSON-KIE at 100% MTP acceptance, 64 tok/s aggregate across 4 concurrent users — and the stress test that breaks the Spark when 100K concurrent contexts hit. Production-relevant memory budget, prefix caching, and an honest closing call: is it worth putting in DocAI?
Qwen3.6-35B-A3B-FP8 + MTP (multi-token prediction) benchmark on DGX Spark, GB10 chip. The vanilla model is just as fast as 3.5, but on the 16-concurrent stress test MTP delivered +24% throughput and −56% TTFT — exactly where spec decoding should have been negative in theory. The unexpected symbiosis of unified memory architecture and speculative decoding.
A two-day vLLM + Triton MoE tuning marathon on DGX Spark with Qwen3.5-35B-A3B-FP8. In the end the production config was 5-7% worse. What I learned about the difference between pure-kernel and serving benchmarks — plus six concrete takeaways you can apply.
The next article is already in the works. If you don’t want to miss it, subscribe through the contact form or check back next week.