The general decision adapter that didn’t generalise
An adapter trained on more than two hundred task families decides more cases — but on tasks it has never seen, it picks an option even when the right answer is “none of these”.
On 9 October we wrote that one recipe, a small decision adapter, automated 13.7 points more invoice lines when matching them to catalogue articles (product matching, from here on), yet did worse than the untrained model in tool selection. The obvious next question: if we train a single adapter on many different tasks, does the gain carry over to a task it has never seen?
We have closed that round, and the answer is no. The adapter makes more decisions everywhere, but on the tasks held out from training it does not reach the precision we set. In product matching and tool selection nearly all its errors have the same cause: the adapter picks one of the options when the correct answer is “none of these”. That is a negative result, and as promised we are publishing it — together with the adapter and its training data.
What we tried
The model is the same as before: Qwen3.6-35B-A3B on vLLM, on our own GB10 machine. The adapter again returns a single decision: the letter of one of 2–10 offered options, or “none of these”. We put a threshold on the probability of that answer; when the model is not sure enough, a person decides.
This time we assembled 17,500 training items from 207 task families. Three quarters of the items come from 21 Hungarian families: processing legislation and procurement notices, reading comprehension, inference, letter triage, recognising and matching entities, judging search results. The remaining quarter are English decision tasks from 186 smaller families. Every item offered “none of these”, and in roughly one in ten it was the right answer, in four types:
- the correct option is removed from the list, but a similar one remains;
- the source itself marks the question as unanswerable;
- the request belongs to a different task altogether;
- a fact needed for the decision is missing.
We used no teacher model, and no closed model’s output went into the training data: every label came from the source or from the structure of the task. Synthetic items make up 16% of the data (for example letters, entities, case types): open-licence models wrote them, and we kept only those where an independent check by a different model family arrived at the same label. On the “none of these” items, an extra term in the training objective pulled the adapter towards the untrained model’s distribution. That was meant to keep the recognition of “none of these” from getting worse.
We measured on three task groups, none of which appeared in training:
- product matching: invoice line → catalogue article, the test from our 9 October post;
- tool selection: which tool an agent should call, if any (BFCL, When2Call, Hungarian requests);
- invoice checking: questions about synthetic invoices (dates, fields, totals).
The yardstick is the same as before: how many cases the system handles automatically if 95% of its decisions must be correct. From here on we call the share of correct decisions among the automated ones precision. The threshold is chosen on a separate validation set of the same kind as the training data. We set the failure line at 93%: if a smaller share of the automated decisions on the test is correct, the adapter has failed. The baseline is the same model without training, carefully calibrated (what that means is explained in the previous post); in the tables it is the calibrated base model. We fixed the hypotheses and decision rules on 14 October, before looking at the adapter’s test results.
The result
| calibrated base model | general adapter | |||
|---|---|---|---|---|
| automated | of which correct | automated | of which correct | |
| product matching | 65.3% | 96.3% | 76.1% | 91.2% |
| tool selection | 64.4% | 91.8% | 73.2% | 86.6% |
| invoice checking | 56.8% | 93.8% | 63.6% | 90.5% |
In all three groups the adapter automates 7–11 points more, but it holds its precision in none of them: 86.6–91.2% of its automated decisions are correct against a 95% target. The adapter failed the pre-registered criterion in all three groups. The figures are averages over three training seeds, and no single seed reached 95%. In tool selection the untrained model is itself below the 93% line; there the question was whether the adapter improves on it.
One pre-registered secondary question formally showed “transfer”: in product matching the general adapter raises the automated share by as much as the earlier, task-specific adapter did. That yardstick looked at the automated share only, not precision: the task-specific adapter reaches that increase at 97% precision, the general one at 91%. So we do not build on it.
Where the errors come from
When a correct option exists, the adapter chooses well. In product matching and tool selection its decisions on such items are 99.1–99.3% correct; the untrained model’s are 99.5–99.7%. So choosing among the options is not what went wrong.
The trouble is “none of these”. Of those items, the adapter assigns 25% in product matching and 28% in tool selection to an option with above-threshold confidence. The untrained model does so for 9% and 15%. And it is not hesitant about it: in tool selection, half of the adapter’s errors come with at least 95% confidence (39% for the untrained model).
Invoice checking is different: there most errors are wrong choices, mainly on questions about the total amount. Those need arithmetic, and without a reasoning step every variant performs poorly on them (59–63% precision).
The gap is widest on genuinely irrelevant requests, where no tool should be called:
| recognising “no suitable tool” | calibrated base model | general adapter |
|---|---|---|
| When2Call | 78% | 54% |
| BFCL-live irrelevance | 66% | 45% |
| BFCL irrelevance | 86% | 67% |
The training data did contain such cases: in a quarter of its “none of these” items, the request belonged to another task. On its own task families the adapter learned this well, raising recognition from 15% to 94%. Yet this did not carry over to the unseen tasks, and the term pulling the adapter towards the untrained model did not prevent the decline. The task-specific tool-selection adapter failed the same way, even though that task was part of its training. This suggests the effect comes not only from how the “none of these” cases in the task-specific training data were constructed, but also from decision training itself.
Why doesn’t the adapter carry over “none of these”?
Our reading: what counts as “none of the options fits” is a convention that differs from domain to domain. The task-specific adapter, trained in its own domain, recognises the “none of these” cases of product matching 94% of the time; the untrained model 42%; the general adapter 34%. What general training teaches is closer to “if there is a similar option, take it”. On familiar tasks that is a good strategy. On unfamiliar ones it produces confident errors.
One more post-hoc observation fits here. A version trained without the English quarter of the data degrades much less on irrelevant English tool requests: it recognises 75% of the When2Call cases instead of 54%. What this shows is that on English tool requests the English decision data pushes the model towards “call the tool”. We saw it on a single training seed and did not test it separately; treat it as a direction, not a finding. On Hungarian tool requests the same data helped (see below).
At equal precision, there is no gain
“More automated decisions” sounds good until both models are held to the same precision. To do that we tuned the threshold on the test itself for 95% precision. That is not an option in production, so the figures below are upper bounds:
| automated at 95% precision | calibrated base model | general adapter |
|---|---|---|
| product matching | 68.2% | 65.0% |
| tool selection | 47.1% | 1.6% |
| invoice checking | 52.5% | 52.8% |
At equal precision the adapter is not meaningfully better anywhere, and in tool selection it can automate practically nothing: its confident errors sit at the very top of its ranking (the decisions ordered by confidence), which puts 95% out of reach. So in product matching and tool selection, the higher automated share in the first table does not come from a better ranking; it comes from a validation threshold that is too permissive on unseen tasks. In invoice checking the adapter’s overall ranking is genuinely better, but it does not show in the 95% precision range.
We also tried what happens if the threshold is chosen on a sample of at most 200 items from the target task. For product matching and tool selection, no threshold on that sample satisfied the threshold-selection rule either, not even for the untrained model: with confident errors at the top of the ranking, that rule is too strict for a sample this small. In invoice checking the method worked, but there the adapter did no better than the untrained model.
Could a two-stage decision save it?
After the fact, using the existing measurements, we tried an obvious fix. The untrained model decides whether “none of these” is the right answer; if it thinks an option fits, the adapter picks which one. The decline disappears: tool selection reaches 91.6% precision instead of 86.6%, the untrained model’s level.
Little gain remains, though. At equal precision the two-stage decision automates about as much as the untrained model: 67.5% against 68.2% in product matching, 49.0% against 47.1% in tool selection. The reason is the ceiling we saw above: in these two groups, when a correct option exists, the untrained model is already almost error-free, which leaves the adapter little to add.
What did general training bring after all?
- Willingness to decide. The adapter decides more cases everywhere. In product matching and tool selection, when a correct option exists, it also decides correctly.
- One type of “none of these” case in invoice checking. When none of the listed invoice lines is wrong, the untrained model recognises it 1% of the time and the adapter 84%. When a fact needed for the decision is missing from the invoice, the adapter is only slightly better: 58% instead of 50%.
- It did not pick up the old, wrong labels. Where we had corrected the source dataset’s labels, the previous post’s task-specific tool-selection adapter confidently returned the old, wrong label (26–29 times out of 51). The general adapter returns the old label about as often as the untrained model: 12–15 times against 13 — though, to be fair, tool selection was not part of its training this time.
- The English data helped on Hungarian tool requests. This is a pre-registered secondary result: on the Hungarian part of tool selection, the version trained with the English data ranks better than the one trained without the English quarter (one training seed each). The same English data made the ranking worse in product matching and made no difference in invoice checking. (On English requests the direction is reversed, see above.)
Release and the next step
Negative results rarely get written up, though they teach just as much. So we are releasing:
- the adapter, with all three training seeds: k3dani/Qwen3.6-35B-A3B-hu-general-decision-lora-negative-result;
- the training and validation data, split by licence group: k3dani/hu-general-decision-mix.
The first paragraph of the model card states plainly that this is a research artefact and is not meant for making decisions. We did not run it on the Decision Index (a public benchmark suite for decision models), and we did not compare it with the open JEV-35B decision model. If you do, we would love to see the results.
The next round is drafted: a separately trainable decision head, splitting the “none of these” output into types, and measuring JEV-35B on our own tests. For now, though, we are stopping here.
In summary
An adapter trained on many tasks does not bring the gains of task-specific adapters to unseen tasks. It carries over the willingness to decide but not the recognition of “none of these”, and the result is confident errors.
- Also measure at equal precision. Our “more automated decisions” came not from a better model but from a validation threshold that was too permissive on unseen tasks.
- Don’t carry a threshold from one task to another. The threshold chosen on validation gave 87–91% precision on unseen tasks instead of 95%.
- Bring labelled examples from the target domain for “none of these”. In our reading that is domain-specific knowledge, and general training does not bring it; the next round would test this.
- Don’t underestimate the untrained, calibrated model. In product matching and tool selection, when a correct option exists, its above-threshold decisions are already 99.5–99.7% correct.
- Fix in advance what counts as failure. After the fact it would have been easy to call “more automated decisions” a success.
This was a pre-registered round: hypotheses, endpoints and decision rules were fixed on 14 October 2026, before the comparative results were viewed. Model: Qwen3.6-35B-A3B-FP8, vLLM 0.30, NVIDIA GB10; LoRA r16, three training seeds; 17,500 training items. Test: 4,524 + 7,416 + 2,200 scored items in the three groups held out from training; the threshold was chosen for 95% target precision by a one-sided Clopper–Pearson rule; clustered bootstrap with 10,000 resamples. The breakdown of errors by source is descriptive, and the two-stage decision and the effect of the English data on English requests are exploratory; all three are post hoc. The protocol, code and aggregate results are in the docai-evals repository; the adapter and the data are on Hugging Face.