of decisions accepted automatically
basal-2.0-max · measured error: 4.51%
BENCHMARK / 11 Oct 2026
Text. Images. Audio.
Results you can verify.
basal-2.0 and basal-omni are compared in the text-only Werdykt benchmark. Images and recordings are evaluated separately in a Polish multimodal probe. Choose a test to see its results, measurement conditions and limitations.
How did we select the models?
We initially selected decision models from the leading entries in Benchmark Heaven (JevBench). We evaluate this selected group in our own Werdykt benchmark, using the same questions. We also include the basal family, large API models and control measurements. Earlier releases remain in the table as reference points.
5 000decisions in Werdykt
10 × 2categories × languages
94systems in the full table
5 probesimages and audio · basal-omni
01 / FULL COMPARISON
Download results JSON Werdykt v1
Werdykt is a benchmark developed by the basal project. We report mean accuracy across 20 groups: 10 categories in two languages. All models, including basal-omni, receive text only here. The table includes completed full-test runs. basal-2.0 uses fast mode (bf16); basal-omni uses eager mode (bf16). Media results are in the “Images and audio” tab.
Engine measurements: basal-2.0 and basal-omni configurations on H100 ↓
Show:
Size counts all model parameters: MoE models (e.g. 26B-A4B, 35B-A3B) belong to the 15–40B class, with active parameters shown in the table. API systems and models without a reported size use the “unknown” class.
94 systems · scroll the table horizontally on smaller screens.
| Rank · model / parameters | PL + EN | 95% interval PL + EN | Polish | English | p50 latency ms ↓ | p95 latency ms ↓ | USD / 1000 decisions ↓ | Engine / reasoning |
|---|---|---|---|---|---|---|---|---|
| 1 Gemini 3.8 Flash— · API | 99.56% | 99.38%–99.72% | 99.56% | 99.56% | 3 314.5 | 7 379.4 | 2.5297 | APIProvider APIReasoning: Provider defaults |
| 2 Claude Opus 5.5†— · API | 99.40% | 99.18%–99.62% | 99.20% | 99.60% | 5 531.3 | 9 362.2 | 15.1738 | APIProvider APIReasoning: Provider defaults |
| 3 GPT-6.1 Sol— · API | 99.38% | 99.16%–99.60% | 99.24% | 99.52% | 2 213.6 | 4 928.0 | 3.1780 | APIProvider APIReasoning: Provider defaults |
| 4 GPT-6 Astra†— · API | 99.34% | 99.12%–99.56% | 99.20% | 99.48% | 1 993.9 | 4 356.5 | 15.8178 | APIProvider APIReasoning: Provider defaults |
| 5 GPT-6 Luna— · API | 98.32% | 97.94%–98.64% | 97.84% | 98.80% | 2 380.1 | 4 210.5 | 0.1880 | APIProvider APIReasoning: Provider defaults |
| 6 Gemini 3.1 Pro (preview)†— · API5 unparsed answers / 5,000 | 97.88% | 97.48%–98.24% | 97.92% | 97.84% | 5 539.5 | 19 820.1 | 12.2981 | APIProvider APIReasoning: Provider defaults |
| 7 René-1 31B FP8¹31B · open weights | 85.36% | 84.48%–86.28% | 85.00% | 85.72% | 139.8 | 1 849.2 | 0.3902 | Model server1 × H100 80 GB |
| 8 pplx-decider-v1.1-27b (Decisions API)¹27B · API | 85.18% | 84.26%–86.08% | 85.04% | 85.32% | 263.5 | 599.4 | 0.0266 | APIProvider API |
| 9 Mercury Decide (OpenRouter Decisions API)¹Not disclosed · API | 84.48% | 83.56%–85.32% | 84.48% | 84.48% | 420.4 | 1 021.4 | 0.0260 | APIProvider API |
| 10 deck-31B¹31B · open weights | 83.54% | 82.54%–84.46% | 83.44% | 83.64% | 1 260.5 | 2 555.1 | 1.5840 | Model server1 × H100 80 GB |
| 11 decisio v0.8.0 on gemma-4-31B-it¹31B · open weights | 83.48% | 82.52%–84.38% | 83.36% | 83.60% | 64.9 | 499.6 | 0.1207 | Model server1 × H100 80 GB |
| 12 Quyet-1.0-Large¹31.3B (30.7B text) · open weights29 request errors / 5,000 | 83.28% | 82.30%–84.20% | 82.80% | 83.76% | 149.0 | 1 333.2 | 0.3093 | Model server1 × H100 80 GB |
| 13 Decider chat · Gemma-4-31B¹31B · open weights | 82.46% | 81.54%–83.40% | 82.48% | 82.44% | 145.6 | 3 406.2 | 0.3003 | Model server1 × RTX PRO 6000 Blackwell 96 GB |
| 14 basal-omni-1.0-max¹12B · open weights | 82.28% | 81.34%–83.24% | 82.44% | 82.12% | 76.2 | 1 359.6 | 0.2631 | basal-serve · eager1 × H100 80 GB |
| 15 Eikos-27B-FP8 (v1.3)†¹27B · open weights | 82.12% | 81.14%–83.10% | 81.92% | 82.32% | 115.8 | 522.9 | 0.0902 | Model server1 × H100 80 GB |
| 16 Jebadiah 27B†¹27B · open weights | 81.96% | 81.02%–82.92% | 82.32% | 81.60% | 96.7 | 843.9 | 0.1970 | Model server1 × H100 80 GB |
| 17 pplx-decider-v1-27b†¹27B · open weights112 request errors / 5,000 | 81.80% | 80.84%–82.70% | 81.84% | 81.76% | 93.0 | 671.8 | 0.1689 | Model server1 × H100 80 GB |
| 18 Surogate Rune 26B-A4B v3¹26B / 4B active · open weights | 81.68% | 80.66%–82.64% | 80.88% | 82.48% | 164.1 | 1 509.9 | 0.1533 | surogate · C++1 × RTX PRO 6000 Blackwell 96 GB |
| 19 Jev 1.13.0— · API | 81.64% | 80.70%–82.58% | 82.04% | 81.24% | 286.4 | 378.8 | 0.0710 | APIProvider API |
| 20 Solar Decide¹— · API | 81.32% | 80.32%–82.28% | 78.52% | 84.12% | 1 271.7 | 28 248.0 | 0.0916 | APIProvider API |
| 21 Liquid d1¹— · API | 80.78% | 79.82%–81.76% | 80.52% | 81.04% | 329.0 | 491.5 | 0.0498 | APIProvider API |
| 22 Clef (OpenRouter Decisions API)¹27B · API | 80.76% | 79.78%–81.72% | 81.28% | 80.24% | 377.1 | 1 199.2 | 0.3345 | APIProvider API |
| 23 Bobcat Flash 1.2¹26B / 4B active · open weights | 80.22% | 79.16%–81.22% | 80.04% | 80.40% | 38.4 | 206.9 | 0.0319 | Model server1 × H100 80 GB |
| 24 simple-jev · Qwen3.8-27B (featherless)†¹27B · open weights | 80.14% | 79.14%–81.08% | 80.00% | 80.28% | 277.3 | 1 024.2 | 0.4017 | Model server1 × H100 80 GB |
| 25 Microsoft-Decision-1 (OpenRouter Decisions API)¹Not disclosed · API | 80.02% | 79.04%–81.02% | 79.04% | 81.00% | 331.7 | 414.7 | 0.0537 | APIProvider API |
| 26 Xor 26B-A4B 1.0¹26B / 4B active · open weights | 79.76% | 78.72%–80.74% | 79.48% | 80.04% | 95.9 | 280.0 | 0.0396 | Model server1 × H100 80 GB |
| 27 Reflex 27B (Decision Index 0.2 build)†¹27B · open weights | 79.10% | 78.04%–80.06% | 79.52% | 78.68% | 135.4 | 500.7 | 0.1133 | Model server1 × H100 80 GB |
| 28 decider-12b v212B · open weights | 78.90% | 77.94%–79.94% | 79.12% | 78.68% | 48.9 | 975.1 | 0.1588 | Model server1 × H100 80 GB |
| 29 Cygnet¹12B · open weights | 78.52% | 77.52%–79.54% | 79.04% | 78.00% | 39.6 | 276.9 | 0.0522 | Model server1 × H100 80 GB |
| 30 Winnow-12B (Q8)¹12B · open weights | 78.50% | 77.42%–79.44% | 78.56% | 78.44% | 126.2 | 1 096.7 | 0.2599 | Model server1 × H100 80 GB |
| 31 basal-2.0-max¹11B · open weights | 78.40% | 77.36%–79.46% | 78.68% | 78.12% | 37.4 | 344.3 | 0.0678 | basal v2.0.0 · fast1 × H100 80 GB |
| 32 GPT-6 Luna Decisions¹— · API | 78.38% | 77.36%–79.38% | 79.24% | 77.52% | 438.0 | 623.3 | 0.1304 | APIProvider API |
| 33 JEVfire (Qwen3.8-27B-FP8)†¹27B · open weights | 77.74% | 76.74%–78.70% | 77.84% | 77.64% | 93.6 | 526.4 | 0.1020 | Model server1 × H100 80 GB |
| 34 basal-1.5-max¹11B · open weights | 77.28% | 76.24%–78.28% | 77.68% | 76.88% | 67.1 | 476.6 | 0.1081 | SGLang1 × H100 80 GB |
| 35 Xor 1.2¹35B / 3B active · open weights | 77.22% | 76.16%–78.20% | 77.08% | 77.36% | 145.9 | 370.0 | 0.0356 | Model server1 × RTX PRO 6000 Blackwell 96 GB |
| 36 decider-12b v112B · open weights | 77.10% | 76.10%–78.14% | 76.88% | 77.32% | 47.6 | 977.7 | 0.1594 | Model server1 × H100 80 GB |
| 37 basal-omni-1.0-mini¹4B · open weights | 76.88% | 75.84%–77.84% | 77.48% | 76.28% | 66.6 | 675.1 | 0.1263 | basal-serve · eager1 × H100 80 GB |
| 38 Decider chat · Qwen3.6-27B¹27B · open weights | 76.86% | 75.76%–77.88% | 76.84% | 76.88% | 90.7 | 1 165.9 | 0.2352 | Model server1 × H100 80 GB |
| 39 Jev-Omni¹12B · open weights | 75.34% | 74.22%–76.38% | 76.32% | 74.36% | 1 131.5 | 1 555.0 | 1.2397 | Model server1 × H100 80 GB |
| 40 H2O-Lightning-4B v1.1¹4B · open weights | 75.30% | 74.20%–76.44% | 74.52% | 76.08% | 58.6 | 127.6 | 0.0206 | Model server1 × H100 80 GB |
| 41 JPT-9B¹9B · open weights | 75.18% | 74.08%–76.24% | 74.12% | 76.24% | 159.5 | 339.2 | 0.0545 | Model server1 × H100 80 GB |
| 42 Decision 1.0 Lux 9B¹9B · open weights | 74.94% | 73.80%–76.06% | 73.84% | 76.04% | 56.6 | 325.9 | 0.0882 | Model server1 × H100 80 GB |
| 43 DREX v1.5 (OpenRouter Decisions API)¹Not disclosed · API | 74.88% | 73.70%–75.96% | 73.60% | 76.16% | 310.8 | 641.1 | 0.0505 | APIProvider API |
| 44 MedDecider-9B¹9.7B · open weights | 73.28% | 72.18%–74.44% | 72.08% | 74.48% | 151.0 | 545.7 | 0.2105 | Model server1 × H100 80 GB |
| 45 basal-2.0¹4.5B · open weights | 72.86% | 71.70%–73.92% | 72.88% | 72.84% | 25.0 | 188.1 | 0.0341 | basal v2.0.0 · fast1 × H100 80 GB |
| 46 basal-1.5¹4.5B · open weights | 72.12% | 71.02%–73.26% | 72.44% | 71.80% | 33.8 | 270.2 | 0.0477 | SGLang1 × H100 80 GB |
| 47 Decision-2.0-Nox-4B¹4B (4.21B) · open weights | 71.64% | 70.46%–72.80% | 70.92% | 72.36% | 863.8 | 1 302.4 | 1.0093 | eager · H100 NVL1 × H100 NVL 94 GB |
| 48 Clef Flash (OpenRouter Decisions API)¹9B · API | 71.60% | 70.44%–72.74% | 70.12% | 73.08% | 345.8 | 700.5 | 0.1254 | APIProvider API |
| 49 Clef-Flash¹9B · open weights | 71.58% | 70.40%–72.70% | 70.08% | 73.08% | 1 700.2 | 1 857.0 | 1.8990 | Model server1 × H100 80 GB |
| 50 JPT-4B¹4B · open weights | 71.14% | 69.98%–72.34% | 70.12% | 72.16% | 69.5 | 193.0 | 0.0412 | Model server1 × H100 80 GB |
| 51 Hopper (G) 1.2¹4B · open weights | 71.10% | 69.96%–72.20% | 70.68% | 71.52% | 59.5 | 169.0 | 0.0694 | Model server1 × H100 80 GB |
| 52 sonda-1.0-4B¹4B · open weights113 request errors / 5,000 | 70.72% | 69.60%–71.90% | 69.92% | 71.52% | 51.4 | 181.6 | 0.0359 | vLLM1 × H100 NVL 94 GB |
| 53 Qwen3-32B, zero-shot (jqv)¹32B · open weights | 70.54% | 69.42%–71.68% | 69.52% | 71.56% | 129.5 | 910.7 | 0.2298 | Model server1 × H100 80 GB |
| 54 decider-35b-a3b NVFP4¹35B / 3B active · open weights | 70.38% | 69.26%–71.54% | 70.20% | 70.56% | 48.0 | 190.7 | 0.0173 | Model server1 × RTX PRO 6000 Blackwell 96 GB |
| 55 Hopper4B · open weights | 70.28% | 69.16%–71.42% | 69.40% | 71.16% | 60.4 | 165.4 | 0.1376 | Model server1 × H100 80 GB |
| 56 Decision 4B v1.2¹4B · open weights | 70.16% | 69.08%–71.34% | 69.84% | 70.48% | 28.7 | 206.9 | 0.0710 | Model server1 × H100 80 GB |
| 57 Clef Omni (OpenRouter Decisions API)¹Not disclosed · API | 70.14% | 68.90%–71.22% | 69.12% | 71.16% | 182.3 | 318.8 | 0.2210 | APIProvider API |
| 58 decider-4b v2¹4B · open weights | 70.00% | 68.86%–71.14% | 69.68% | 70.32% | 27.8 | 283.3 | 0.0515 | Model server1 × H100 80 GB |
| 59 djev¹26B / 4B active · open weights274 request errors / 5,000 | 69.80% | 68.72%–70.98% | 70.16% | 69.44% | 66.6 | 159.1 | 0.0526 | Model server1 × H100 80 GB |
| 60 spark-s1-4b-v6¹4B · open weights | 69.38% | 68.16%–70.58% | 68.84% | 69.92% | 217.4 | 565.2 | 0.2879 | open_spark_jev · HF1 × H100 80 GB |
| 61 Plumb-4B¹4B · open weights | 69.04% | 67.88%–70.32% | 68.92% | 69.16% | 27.7 | 245.7 | 0.0553 | Model server1 × H100 80 GB |
| 62 Decision 4B v1.1¹4B · open weights | 69.02% | 67.88%–70.22% | 68.76% | 69.28% | 28.5 | 207.1 | 0.1236 | Model server1 × H100 80 GB |
| 63 MedDecider-4B¹4.7B · open weights | 68.90% | 67.80%–70.18% | 68.44% | 69.36% | 151.1 | 418.4 | 0.1939 | Model server1 × H100 80 GB |
| 64 JevK5 v0.3¹4B · open weights | 68.86% | 67.68%–70.12% | 68.28% | 69.44% | 28.4 | 258.5 | 0.0570 | Model server1 × H100 80 GB |
| 65 Tev1 4B Experimental¹4B · API | 68.62% | 67.46%–69.78% | 68.04% | 69.20% | 239.6 | 350.2 | 0.0569 | APIProvider API |
| 66 Kev 1.0 (kev-4b)¹4B · open weights | 68.36% | 67.14%–69.64% | 67.16% | 69.56% | 34.1 | 308.9 | 0.0442 | kev.serve1 × H100 80 GB |
| 67 Imajev-4B¹4B · open weights436 request errors / 5,000 | 68.16% | 67.10%–69.24% | 66.56% | 69.76% | 112.7 | 132.3 | 0.1349 | Model server1 × H100 80 GB |
| 68 Kev 4B (OpenRouter Decisions API)¹4B · API108 request errors / 5,000 | 67.44% | 66.24%–68.72% | 66.00% | 68.88% | 878.8 | 1 362.7 | 0.0442 | APIProvider API |
| 69 Jet v6.2¹4B · open weights | 67.34% | 66.12%–68.50% | 66.76% | 67.92% | 934.5 | 1 063.7 | 0.3490 | Model server1 × H100 80 GB |
| 70 JevK5 v0.2 (4B)¹4B · open weights | 66.72% | 65.56%–67.94% | 66.40% | 67.04% | 30.7 | 240.4 | 0.0562 | jevk5-serve1 × H100 80 GB |
| 71 basal-1.0 · 4.5B4.5B · open weights | 66.04% | 64.70%–67.18% | 66.20% | 65.88% | 26.5 | 779.5 | 0.1353 | basal · fast1 × H100 80 GB |
| 72 sonda-1.1-4B¹4B · open weights | 64.02% | 62.80%–65.26% | 63.92% | 64.12% | 67.2 | 136.6 | 0.0252 | Model server1 × H100 80 GB |
| 73 Malkuth-4B¹4B · open weights108 request errors / 5,000 | 63.36% | 62.10%–64.66% | 62.24% | 64.48% | 153.2 | 285.2 | 0.1722 | Model server1 × H100 80 GB |
| 74 Manchego v2.14B · open weights | 63.04% | 61.76%–64.30% | 62.20% | 63.88% | 73.5 | 400.8 | 0.1149 | Model server1 × H100 80 GB |
| 75 local-jev (Qwen3.5-4B)¹4B · open weights | 62.98% | 61.76%–64.16% | 62.48% | 63.48% | 142.7 | 535.8 | 0.2534 | local-jev · HF1 × H100 80 GB |
| 76 typecastlm v1.3.0 (Qwen3.5 3.8B)¹3.8B · open weights | 62.88% | 61.72%–64.14% | 62.36% | 63.40% | 95.2 | 217.3 | 0.1381 | typecastlm · local1 × H100 80 GB |
| 77 lev4B · open weights | 62.86% | 61.64%–64.12% | 61.68% | 64.04% | 157.6 | 538.1 | 0.2013 | Model server1 × H100 80 GB |
| 78 jev-local (Qwen3.5-9B, HF scorer)¹9B · open weights436 request errors / 5,000 | 62.18% | 61.08%–63.32% | 61.56% | 62.80% | 460.7 | 829.0 | 2.0440 | jevlocal · HF scorer1 × H100 80 GB |
| 79 Qwen3-4B-Instruct-2507 (raw, letter logits)¹4B · open weights · control measurement | 62.08% | 60.90%–63.22% | 61.40% | 62.76% | 40.0 | 161.2 | 0.0222 | vLLM · letter logits1 × H100 80 GB |
| 80 basal-2.0-mini¹1.5B · open weights | 61.84% | 60.54%–63.08% | 62.76% | 60.92% | 15.5 | 93.9 | 0.0103 | basal v2.0.0 · fast1 × H100 80 GB |
| 81 metask-jev-4b¹4B · open weights441 request errors / 5,000 | 61.16% | 59.90%–62.34% | 60.68% | 61.64% | 107.8 | 177.1 | 0.3400 | Model server1 × H100 80 GB |
| 82 basal-1.5-mini1.5B · open weights | 60.66% | 59.40%–61.92% | 61.92% | 59.40% | 14.2 | 346.4 | 0.0535 | basal · fast1 × H100 80 GB |
| 83 open-alternative-jev (so1, Qwen3.5-4B)¹4B · open weights | 60.24% | 58.94%–61.50% | 59.16% | 61.32% | 101.4 | 433.4 | 0.1579 | so1 · HF + HTTP adapter1 × H100 80 GB |
| 84 Solar Decide Flash¹— · API | 60.16% | 58.92%–61.42% | 57.12% | 63.20% | 940.7 | 1 684.6 | 0.0916 | APIProvider API |
| 85 SemIf¹4B · open weights436 request errors / 5,000 | 58.80% | 57.62%–59.92% | 57.52% | 60.08% | 66.0 | 119.4 | 0.0800 | Model server1 × H100 80 GB |
| 86 decider-2b (v11)¹2B · open weights | 58.66% | 57.40%–59.94% | 58.04% | 59.28% | 20.3 | 168.5 | 0.0248 | decider-ai 1.8.11 × H100 80 GB |
| 87 JEV-Qwen3.5-9B-Base-NVFP4¹9B · open weights | 57.26% | 55.92%–58.52% | 57.76% | 56.76% | 31.8 | 177.9 | 0.0264 | SGLang · NVFP4 + HTTP adapter1 × RTX PRO 6000 Blackwell 96 GB |
| 88 Phi-4-mini-instruct (raw, letter logits)¹3.8B · open weights · control measurement | 55.78% | 54.40%–57.02% | 53.68% | 57.88% | 35.9 | 132.9 | 0.0171 | vLLM · letter logits1 × H100 80 GB |
| 89 Decision 2B (preview)¹2B · open weights431 request errors / 5,000 | 54.62% | 53.36%–55.78% | 49.24% | 60.00% | 48.7 | 76.8 | 0.0580 | model.decide + HTTP adapter1 × H100 80 GB |
| 90 basal-1.0 · 1.5B1.5B · open weights | 52.24% | 50.94%–53.62% | 52.72% | 51.76% | 14.6 | 354.4 | 0.0549 | basal · fast1 × H100 80 GB |
| 91 Nemotron Diffusion 8B (Nemotron_Jev)¹8B · open weights | 51.54% | 50.16%–52.94% | 48.72% | 54.36% | 38.6 | 210.7 | 0.0628 | Nemotron_Jev · vLLM1 × H100 80 GB |
| 92 Malkuth-2B¹2B · open weights108 request errors / 5,000 | 49.98% | 48.74%–51.26% | 48.16% | 51.80% | 73.2 | 119.9 | 0.0765 | kev.serve1 × H100 80 GB |
| 93 Qwen3-Reranker-4B (as a decider)¹4B · open weights · control measurement | 47.20% | 46.02%–48.44% | 45.92% | 48.48% | 131.6 | 1 089.1 | 0.2743 | Transformers · reranker adapter1 × H100 80 GB |
| 94 Certo v1¹0.4B · open weights | 28.26% | 27.04%–29.38% | 29.40% | 27.12% | 20.8 | 25.0 | 0.0238 | certo · HTTP adapter1 × H100 80 GB |
† Result is not independent. The model, or a model from the same family, helped create or check the questions. These results should not be treated as an independent comparison.
¹ Configuration differences. Cygnet: field-name adapter and a 16,384-token limit; longer inputs rejected by the server count as errors. basal-1.5 and max on SGLang: 32,768 context limit instead of the default 4,096. basal-2.0 and basal-omni-1.0: answers from --deterministic mode (answers independent of load), with time and cost from a separate run in default mode on the same server; basal-omni-1.0 uses eager mode. Werdykt measures text only: basal-omni (the experimental Gemma 4-based line, mini 4B and max 12B) also accepts images and audio, which this table does not evaluate (image and audio results). Winnow: 65,536 context as specified in its README. Nox: H100 NVL 94 GB and eager mode. Provider decision interfaces (Decisions API through OpenRouter) receive each question as a single choice question; we select the option with the highest returned probability. Some open models without their own HTTP server run through our thin adapter, which does not change model answers. Measurements come from different cloud environments. Each system lists its GPU.
Blackwell and H100. Systems marked “RTX PRO 6000 Blackwell 96 GB” run on that GPU, following the authors' configuration or because they do not fit on one H100. Their time and cost refer to different hardware from the other rows.
Control measurements. Qwen3-4B-Instruct-2507 and Phi-4-mini-instruct are general-purpose models evaluated using answer-letter probabilities. Qwen3-Reranker-4B scores options through a reranker adapter and truncates long inputs to 8,192 tokens. These three rows are reference points in Werdykt, not specialised decision models.
Execution limitations. Request errors and unparsed responses count as incorrect decisions; counts appear by the model names. Decision 2B rejected 431 questions due to length limits, Malkuth-2B rejected 108, and jev-local (Qwen3.5-9B, HF scorer) rejected 436 due to its 4,096-token limit. jev-local p50 and p95 latency use 181 successfully processed requests from a 200-question sample. Certo reads only the first 64 tokens of the question and state. Library timings include our HTTP adapter where no native server exists. Spark, local-jev and so1 use the reference Transformers path without FLA; Spark also falls back to a full pass after a cache error. JevK5 v0.2 uses reference convolution. Installing CPU dependencies during part of the decider-2b measurement may have slightly reduced its throughput.
Nox protocol. The full 5,000-question result is in the main table. The model uses eager mode on H100 NVL 94 GB; its fast path is validated by its authors for ROCm. Latency and throughput were measured with one server process. The remaining 3,000 questions used a pool of five processes solely to assess accuracy; their timings were not used to calculate cost or speed.
How we calculate cost. H100 and H100 NVL: reference rate of USD 3.95/h; RTX PRO 6000 Blackwell: USD 2.09/h. Cost per 1,000 decisions is derived from throughput at full load. APIs: cost reported by the provider. A lightly loaded deployment may cost more per decision. “—” means data is unavailable.
Accuracy vs response time
Higher and further left means better accuracy with less waiting. The time axis is logarithmic; the table provides exact values.
basal familyOther open modelsAPI
basal-2.0 and basal-omni configurations
Three text models and two multimodal models on H100 80 GB. Each received the same 5,000 Werdykt text questions. Below are both families' configurations and results from the full table above.
| Model / parameters | Engine · BF16 | Hardware | PL + EN ↑ | p50 latency ms ↓ | p95 latency ms ↓ | USD / 1000 decisions ↓ |
|---|---|---|---|---|---|---|
| basal-2.0-mini1.5B | basal v2.0.0 · fast | 1 × H100 80 GB | 61.84% | 15.5 | 93.9 | 0.0103 |
| basal-2.04.5B | basal v2.0.0 · fast | 1 × H100 80 GB | 72.86% | 25.0 | 188.1 | 0.0341 |
| basal-2.0-max11B | basal v2.0.0 · fast | 1 × H100 80 GB | 78.40% | 37.4 | 344.3 | 0.0678 |
| basal-omni-1.0-mini4B | basal-serve · eager | 1 × H100 80 GB | 76.88% | 66.6 | 675.1 | 0.1263 |
| basal-omni-1.0-max12B | basal-serve · eager | 1 × H100 80 GB | 82.28% | 76.2 | 1 359.6 | 0.2631 |
Text in both families. basal-2.0 uses fast mode; basal-omni uses eager mode. All measurements use BF16, one option order and enabled calibration. Accuracy, time and cost here refer only to text. See separate basal-omni image and audio results →
Measurement conditions. Accuracy comes from all 5,000 questions in --deterministic mode. p50 and p95 latency were measured on 200 shared questions, one request at a time. Time and throughput come from a separate run in default mode on the same server used to measure each model's accuracy.
Cost at a shared reference rate. All costs are derived from throughput with 16 concurrent requests and a reference rate of USD 3.95/h. This is a benchmark comparison rate, not the rental price of this machine.
BASAL-OMNI / IMAGES AND AUDIOThe same decision.
Download results JSON ↓The same decision.
A different source of information.
This is a separate, small Polish probe, omni-probe-pl: five tasks and 595 questions per model. Each question was asked with an image or recording and with its text equivalent. Werdykt in the first tab measures text only — including for omni models.
Experimental
basal-omni-1.0-mini (4B) and max (12B) were trained on text decisions. Gemma 4 encoders handle media without additional multimodal training (zero-shot). Confidence thresholds were fitted on text, so they need validation on your own media.
What do these measurements show?
Max reads document images better: 83.8% vs mini's 72.5%. Both lose some accuracy compared with receiving ready-to-use document text.
Mini handles recordings better: 92.1% intent recognition vs max's 71.4%. If you have a transcript, compare its results with direct audio input.
Photos help compared with the question alone, but these scores are close to the base model without basal training. We do not treat them as evidence of a general multimodal advantage.
These are small probes of different difficulty. We do not combine them into one ranking or merge them with Werdykt scores.
omni-mini · 4B omni-max · 12B
Document image
Image · 160 questions
Real photo
Image · 150 questions
Intent of a recorded request · MINDS-14
Audio · 140 questions
Recorded excerpt topic
Audio · 100 questions
Decision from a recorded statement
Audio · 45 questions
Media, text and base model — all results
Document images are compared with their text content; recordings with transcripts. For photos, the text baseline contains only the question, without information from the photo. It is therefore not an equivalent image description.
| Task / questions | Model | From media ↑ | From text ↑ | Gemma 4 without basal training from the same media ↑ | What does the text input contain? |
|---|---|---|---|---|---|
| Document image160 questions · Image | omni-mini | 72.5% | 83.1% | 50.0%Gemma 4 E4B | Document content |
| Document image160 questions · Image | omni-max | 83.8% | 90.0% | 67.5%Gemma 4 12B | Document content |
| Real photo150 questions · Image | omni-mini | 66.7% | 50.7% | 64.0%Gemma 4 E4B | Question only, without the image |
| Real photo150 questions · Image | omni-max | 66.0% | 52.7% | 68.0%Gemma 4 12B | Question only, without the image |
| Intent of a recorded request · MINDS-14140 questions · Audio | omni-mini | 92.1% | 94.3% | 90.0%Gemma 4 E4B | Transcript |
| Intent of a recorded request · MINDS-14140 questions · Audio | omni-max | 71.4% | 92.9% | 70.0%Gemma 4 12B | Transcript |
| Recorded excerpt topic100 questions · Audio | omni-mini | 87.0% | 90.0% | 86.0%Gemma 4 E4B | Transcript |
| Recorded excerpt topic100 questions · Audio | omni-max | 51.0% | 93.0% | 55.0%Gemma 4 12B | Transcript |
| Decision from a recorded statement45 questions · Audio | omni-mini | 84.4% | 95.6% | 60.0%Gemma 4 E4B | Transcript |
| Decision from a recorded statement45 questions · Audio | omni-max | 55.6% | 95.6% | 57.8%Gemma 4 12B | Transcript |
Scope. 160 questions about rendered document images, 150 about real photos, 140 about intent in recorded requests (MINDS-14), 100 about recording topics and 45 about decisions based on speech. The same set for both models. Results are rounded as in the model cards; differences of a few points in such small probes may be due to chance.
Separate media and mixed inputs. The results above concern one medium at a time. In a separate probe of 20 document-image and recorded-request pairs, adding the second medium changed 5–15% of mini's answers and 10–25% of max's. The most stable approach is to send each question only with the media it concerns.
Sources. Aggregates from the model cards: basal-omni-1.0-mini · basal-omni-1.0-max. The JSON export includes SHA-256 hashes of the sources used. More about the protocol, setup and limitations: release article.
Time and cost. This set does not publish comparable media latency or cost measurements. We do not transfer speed results from text-only Werdykt here.
02 / POLISH ITEMS ONLY
How do the models perform in Polish?
Ranking of 88 systems from the Werdykt export dated 11 October 2026. Accuracy is the mean across 10 Polish categories, comprising 2,500 questions. Time is the median for the entire Werdykt, not a separate measurement of Polish questions. General-purpose models and the reranker are marked as controls.
72.9% basal-2.078.7% basal-2.0-max62.8% basal-2.0-mini
| Rank | Model | Parameters | PL accuracy ↑ | Median latency ↓ |
|---|---|---|---|---|
| 1 | pplx-decider-v1.1-27b (Decisions API)Provider API | 27B | 85.0% | 263.5 ms |
| 2 | René-1 31B FP81 × H100 80 GB | 31B | 85.0% | 139.8 ms |
| 3 | Mercury Decide (OpenRouter Decisions API)Provider API | Not disclosed | 84.5% | 420.4 ms |
| 4 | deck-31B1 × H100 80 GB | 31B | 83.4% | 1.3 s |
| 5 | decisio v0.8.0 on gemma-4-31B-it1 × H100 80 GB | 31B | 83.4% | 64.9 ms |
| 6 | Quyet-1.0-Large1 × H100 80 GBRequest errors in the full test: 29 / 5 000 | 31.3B (30.7B text) | 82.8% | 149.0 ms |
| 7 | Decider chat · Gemma-4-31B1 × RTX PRO 6000 Blackwell 96 GB | 31B | 82.5% | 145.6 ms |
| 8 | basal-omni-1.0-max1 × H100 80 GB | 12B | 82.4% | 76.2 ms |
| 9 | Jebadiah 27B1 × H100 80 GB | 27B | 82.3% | 96.7 ms |
| 10 | Jev 1.13.0Provider API | — | 82.0% | 286.4 ms |
| 11 | Eikos-27B-FP8 (v1.3)1 × H100 80 GB | 27B | 81.9% | 115.8 ms |
| 12 | pplx-decider-v1-27b1 × H100 80 GBRequest errors in the full test: 112 / 5 000 | 27B | 81.8% | 93.0 ms |
| 13 | Clef (OpenRouter Decisions API)Provider API | 27B | 81.3% | 377.1 ms |
| 14 | Surogate Rune 26B-A4B v31 × RTX PRO 6000 Blackwell 96 GB | 26B-A4B | 80.9% | 164.1 ms |
| 15 | Liquid d1Provider API | — | 80.5% | 329.0 ms |
| 16 | Bobcat Flash 1.21 × H100 80 GB | 26B-A4B | 80.0% | 38.4 ms |
| 17 | simple-jev · Qwen3.8-27B (featherless)1 × H100 80 GB | 27B | 80.0% | 277.3 ms |
| 18 | Reflex 27B (Decision Index 0.2 build)1 × H100 80 GB | 27B | 79.5% | 135.4 ms |
| 19 | Xor 26B-A4B 1.01 × H100 80 GB | 26B-A4B | 79.5% | 95.9 ms |
| 20 | GPT-6 Luna DecisionsProvider API | — | 79.2% | 438.0 ms |
| 21 | decider-12b v21 × H100 80 GB | 12B | 79.1% | 48.9 ms |
| 22 | Microsoft-Decision-1 (OpenRouter Decisions API)Provider API | Not disclosed | 79.0% | 331.7 ms |
| 23 | Cygnet1 × H100 80 GB | 12B | 79.0% | 39.6 ms |
| 24 | basal-2.0-max1 × H100 80 GB | 11B | 78.7% | 37.4 ms |
| 25 | Winnow-12B (Q8)1 × H100 80 GB | 12B | 78.6% | 126.2 ms |
| 26 | Solar DecideProvider API | — | 78.5% | 1.3 s |
| 27 | JEVfire (Qwen3.8-27B-FP8)1 × H100 80 GB | 27B | 77.8% | 93.6 ms |
| 28 | basal-1.5-max1 × H100 80 GB | 11B | 77.7% | 67.1 ms |
| 29 | basal-omni-1.0-mini1 × H100 80 GB | 4B | 77.5% | 66.6 ms |
| 30 | Xor 1.21 × RTX PRO 6000 Blackwell 96 GB | 35B-A3B | 77.1% | 145.9 ms |
| 31 | decider-12b v11 × H100 80 GB | 12B | 76.9% | 47.6 ms |
| 32 | Decider chat · Qwen3.6-27B1 × H100 80 GB | 27B | 76.8% | 90.7 ms |
| 33 | Jev-Omni1 × H100 80 GB | 12B | 76.3% | 1.1 s |
| 34 | H2O-Lightning-4B v1.11 × H100 80 GB | 4B | 74.5% | 58.6 ms |
| 35 | JPT-9B1 × H100 80 GB | 9B | 74.1% | 159.5 ms |
| 36 | Decision 1.0 Lux 9B1 × H100 80 GB | 9B | 73.8% | 56.6 ms |
| 37 | DREX v1.5 (OpenRouter Decisions API)Provider API | Not disclosed | 73.6% | 310.8 ms |
| 38 | basal-2.01 × H100 80 GB | 4.5B | 72.9% | 25.0 ms |
| 39 | basal-1.51 × H100 80 GB | 4.5B | 72.4% | 33.8 ms |
| 40 | MedDecider-9B1 × H100 80 GB | 9.7B | 72.1% | 151.0 ms |
| 41 | Decision-2.0-Nox-4B1 × H100 NVL 94 GB | 4B (4.21B) | 70.9% | 863.8 ms |
| 42 | Hopper (G) 1.21 × H100 80 GB | 4B | 70.7% | 59.5 ms |
| 43 | decider-35b-a3b NVFP41 × RTX PRO 6000 Blackwell 96 GB | 35B-A3B | 70.2% | 48.0 ms |
| 44 | djev1 × H100 80 GBRequest errors in the full test: 274 / 5 000 | 26B (4B active) | 70.2% | 66.6 ms |
| 45 | Clef Flash (OpenRouter Decisions API)Provider API | 9B | 70.1% | 345.8 ms |
| 46 | JPT-4B1 × H100 80 GB | 4B | 70.1% | 69.5 ms |
| 47 | Clef-Flash1 × H100 80 GB | 9B | 70.1% | 1.7 s |
| 48 | sonda-1.0-4B1 × H100 NVL 94 GBRequest errors in the full test: 113 / 5 000 | 4B | 69.9% | 51.4 ms |
| 49 | Decision 4B v1.21 × H100 80 GB | 4B | 69.8% | 28.7 ms |
| 50 | decider-4b v21 × H100 80 GB | 4B | 69.7% | 27.8 ms |
| 51 | Qwen3-32B, zero-shot (jqv)1 × H100 80 GB | 32B | 69.5% | 129.5 ms |
| 52 | Hopper1 × H100 80 GB | 4B | 69.4% | 60.4 ms |
| 53 | Clef Omni (OpenRouter Decisions API)Provider API | Not disclosed | 69.1% | 182.3 ms |
| 54 | Plumb-4B1 × H100 80 GB | 4B | 68.9% | 27.7 ms |
| 55 | spark-s1-4b-v61 × H100 80 GB | 4B | 68.8% | 217.4 ms |
| 56 | Decision 4B v1.11 × H100 80 GB | 4B | 68.8% | 28.5 ms |
| 57 | MedDecider-4B1 × H100 80 GB | 4.7B | 68.4% | 151.1 ms |
| 58 | JevK5 v0.31 × H100 80 GB | 4B | 68.3% | 28.4 ms |
| 59 | Tev1 4B ExperimentalProvider API | 4B | 68.0% | 239.6 ms |
| 60 | Kev 1.0 (kev-4b)1 × H100 80 GB | 4B | 67.2% | 34.1 ms |
| 61 | Jet v6.21 × H100 80 GB | 4B | 66.8% | 934.5 ms |
| 62 | Imajev-4B1 × H100 80 GBRequest errors in the full test: 436 / 5 000 | 4B | 66.6% | 112.7 ms |
| 63 | JevK5 v0.2 (4B)1 × H100 80 GB | 4B | 66.4% | 30.7 ms |
| 64 | basal-1.0 · 4.5B1 × H100 80 GB | 4.5B | 66.2% | 26.5 ms |
| 65 | Kev 4B (OpenRouter Decisions API)Provider APIRequest errors in the full test: 108 / 5 000 | 4B | 66.0% | 878.8 ms |
| 66 | sonda-1.1-4B1 × H100 80 GB | 4B | 63.9% | 67.2 ms |
| 67 | basal-2.0-mini1 × H100 80 GB | 1.5B | 62.8% | 15.5 ms |
| 68 | local-jev (Qwen3.5-4B)1 × H100 80 GB | 4B | 62.5% | 142.7 ms |
| 69 | typecastlm v1.3.0 (Qwen3.5 3.8B)1 × H100 80 GB | 3.8B | 62.4% | 95.2 ms |
| 70 | Malkuth-4B1 × H100 80 GBRequest errors in the full test: 108 / 5 000 | 4B | 62.2% | 153.2 ms |
| 71 | Manchego v2.11 × H100 80 GB | 4B | 62.2% | 73.5 ms |
| 72 | basal-1.5-mini1 × H100 80 GB | 1.5B | 61.9% | 14.2 ms |
| 73 | lev1 × H100 80 GB | 4B | 61.7% | 157.6 ms |
| 74 | jev-local (Qwen3.5-9B, HF scorer)1 × H100 80 GBRequest errors in the full test: 436 / 5 000 | 9B | 61.6% | 460.7 ms |
| 75 | Qwen3-4B-Instruct-2507 (raw, letter logits)Control measurement1 × H100 80 GB | 4B | 61.4% | 40.0 ms |
| 76 | metask-jev-4b1 × H100 80 GBRequest errors in the full test: 441 / 5 000 | 4B | 60.7% | 107.8 ms |
| 77 | open-alternative-jev (so1, Qwen3.5-4B)1 × H100 80 GB | 4B | 59.2% | 101.4 ms |
| 78 | decider-2b (v11)1 × H100 80 GB | 2B | 58.0% | 20.3 ms |
| 79 | JEV-Qwen3.5-9B-Base-NVFP41 × RTX PRO 6000 Blackwell 96 GB | 9B | 57.8% | 31.8 ms |
| 80 | SemIf1 × H100 80 GBRequest errors in the full test: 436 / 5 000 | 4B | 57.5% | 66.0 ms |
| 81 | Solar Decide FlashProvider API | — | 57.1% | 940.7 ms |
| 82 | Phi-4-mini-instruct (raw, letter logits)Control measurement1 × H100 80 GB | 3.8B | 53.7% | 35.9 ms |
| 83 | basal-1.0 · 1.5B1 × H100 80 GB | 1.5B | 52.7% | 14.6 ms |
| 84 | Decision 2B (preview)1 × H100 80 GBRequest errors in the full test: 431 / 5 000 | 2B | 49.2% | 48.7 ms |
| 85 | Nemotron Diffusion 8B (Nemotron_Jev)1 × H100 80 GB | 8B | 48.7% | 38.6 ms |
| 86 | Malkuth-2B1 × H100 80 GBRequest errors in the full test: 108 / 5 000 | 2B | 48.2% | 73.2 ms |
| 87 | Qwen3-Reranker-4B (as a decider)Control measurement1 × H100 80 GB | 4B | 45.9% | 131.6 ms |
| 88 | Certo v11 × H100 80 GB | 0.4B | 29.4% | 20.8 ms |
For comparison: large models through APIs
A different trade-off between accuracy and response time. † Means the model or its family helped create or check the questions; this result is not an independent comparison.
| Model | PL accuracy ↑ | Median latency ↓ |
|---|---|---|
| Gemini 3.8 Flash | 99.6% | 3.3 s |
| GPT-6.1 Sol | 99.2% | 2.2 s |
| Claude Opus 5.5 † | 99.2% | 5.5 s |
| GPT-6 Astra † | 99.2% | 2.0 s |
| Gemini 3.1 Pro (preview) † | 97.9% | 5.5 s |
| GPT-6 Luna | 97.8% | 2.4 s |
Source: Polish questions from the full test, Werdykt export dated 11 October 2026. Nox was measured on H100 NVL 94 GB, systems marked “RTX PRO 6000 Blackwell” on that GPU, and other open models on H100 80 GB. Interpret differences below roughly 2 percentage points in overall Polish accuracy with caution.
03 / POLISH, CATEGORY BY CATEGORY
One average does not tell the whole story.
Compare 88 systems across 10 decision categories. Each Polish category contains about 250 questions. Colour helps read the scores, and every cell includes the exact value.
Scroll horizontally to see all models. Results are for Polish only.
| Category | pplx-decider-v1.1-27b (Decisions API) | René-1 31B FP8 | Mercury Decide (OpenRouter Decisions API) | deck-31B | decisio v0.8.0 on gemma-4-31B-it | Quyet-1.0-Large | Decider chat · Gemma-4-31B | basal-omni-1.0-max | Jebadiah 27B | Jev 1.13.0 | Eikos-27B-FP8 (v1.3) | pplx-decider-v1-27b | Clef (OpenRouter Decisions API) | Surogate Rune 26B-A4B v3 | Liquid d1 | Bobcat Flash 1.2 | simple-jev · Qwen3.8-27B (featherless) | Reflex 27B (Decision Index 0.2 build) | Xor 26B-A4B 1.0 | GPT-6 Luna Decisions | decider-12b v2 | Microsoft-Decision-1 (OpenRouter Decisions API) | Cygnet | basal-2.0-max | Winnow-12B (Q8) | Solar Decide | JEVfire (Qwen3.8-27B-FP8) | basal-1.5-max | basal-omni-1.0-mini | Xor 1.2 | decider-12b v1 | Decider chat · Qwen3.6-27B | Jev-Omni | H2O-Lightning-4B v1.1 | JPT-9B | Decision 1.0 Lux 9B | DREX v1.5 (OpenRouter Decisions API) | basal-2.0 | basal-1.5 | MedDecider-9B | Decision-2.0-Nox-4B | Hopper (G) 1.2 | decider-35b-a3b NVFP4 | djev | Clef Flash (OpenRouter Decisions API) | JPT-4B | Clef-Flash | sonda-1.0-4B | Decision 4B v1.2 | decider-4b v2 | Qwen3-32B, zero-shot (jqv) | Hopper | Clef Omni (OpenRouter Decisions API) | Plumb-4B | spark-s1-4b-v6 | Decision 4B v1.1 | MedDecider-4B | JevK5 v0.3 | Tev1 4B Experimental | Kev 1.0 (kev-4b) | Jet v6.2 | Imajev-4B | JevK5 v0.2 (4B) | basal-1.0 · 4.5B | Kev 4B (OpenRouter Decisions API) | sonda-1.1-4B | basal-2.0-mini | local-jev (Qwen3.5-4B) | typecastlm v1.3.0 (Qwen3.5 3.8B) | Malkuth-4B | Manchego v2.1 | basal-1.5-mini | lev | jev-local (Qwen3.5-9B, HF scorer) | Qwen3-4B-Instruct-2507 (raw, letter logits) | metask-jev-4b | open-alternative-jev (so1, Qwen3.5-4B) | decider-2b (v11) | JEV-Qwen3.5-9B-Base-NVFP4 | SemIf | Solar Decide Flash | Phi-4-mini-instruct (raw, letter logits) | basal-1.0 · 1.5B | Decision 2B (preview) | Nemotron Diffusion 8B (Nemotron_Jev) | Malkuth-2B | Qwen3-Reranker-4B (as a decider) | Certo v1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Insufficient informationabstain | 78.8% | 76.0% | 84.4% | 80.0% | 80.4% | 77.2% | 78.0% | 80.8% | 83.6% | 73.2% | 84.8% | 88.8% | 82.4% | 76.8% | 79.2% | 72.8% | 78.8% | 81.2% | 75.2% | 80.0% | 70.8% | 73.6% | 70.4% | 80.0% | 70.8% | 79.6% | 72.0% | 73.6% | 74.8% | 66.0% | 70.4% | 69.6% | 68.0% | 69.2% | 67.2% | 64.8% | 75.2% | 69.6% | 70.0% | 56.4% | 63.2% | 59.2% | 56.4% | 57.2% | 58.8% | 61.6% | 59.2% | 60.0% | 69.6% | 60.8% | 61.6% | 57.2% | 64.4% | 63.6% | 60.0% | 69.2% | 53.6% | 57.6% | 57.6% | 59.6% | 51.2% | 74.0% | 57.6% | 59.2% | 59.2% | 50.4% | 54.8% | 53.2% | 48.4% | 49.6% | 44.8% | 52.0% | 51.6% | 52.0% | 51.2% | 54.0% | 51.2% | 48.0% | 51.2% | 51.6% | 50.8% | 45.2% | 41.6% | 42.0% | 34.8% | 36.4% | 28.8% | 21.2% |
| Agent’s next stepaction | 91.2% | 90.8% | 86.8% | 88.0% | 88.0% | 88.8% | 86.0% | 83.2% | 82.0% | 83.6% | 79.6% | 82.8% | 79.2% | 85.2% | 82.8% | 81.6% | 74.8% | 76.4% | 83.2% | 79.6% | 81.6% | 78.4% | 83.6% | 72.0% | 82.4% | 84.4% | 74.4% | 72.8% | 76.4% | 72.4% | 79.6% | 75.6% | 76.4% | 73.6% | 65.2% | 67.6% | 69.6% | 68.8% | 62.4% | 72.4% | 70.0% | 65.6% | 67.2% | 70.8% | 67.2% | 67.6% | 66.4% | 68.4% | 61.2% | 66.0% | 67.2% | 64.8% | 70.8% | 60.4% | 60.4% | 61.2% | 66.8% | 61.6% | 63.2% | 65.6% | 65.6% | 64.0% | 60.0% | 62.8% | 64.8% | 57.2% | 50.4% | 60.8% | 52.4% | 61.6% | 59.6% | 42.0% | 60.4% | 60.0% | 55.6% | 64.8% | 52.0% | 51.6% | 50.8% | 58.4% | 45.6% | 46.4% | 45.6% | 45.6% | 40.8% | 43.6% | 43.6% | 19.2% |
| Contractscontract | 77.6% | 80.8% | 77.2% | 78.8% | 80.4% | 74.4% | 77.6% | 78.4% | 73.6% | 76.0% | 76.8% | 76.4% | 74.4% | 76.0% | 72.0% | 73.6% | 73.2% | 72.0% | 73.2% | 73.2% | 73.2% | 72.8% | 75.2% | 70.8% | 71.6% | 64.8% | 73.6% | 69.6% | 66.0% | 72.8% | 70.8% | 71.2% | 69.6% | 66.8% | 65.6% | 68.4% | 62.4% | 65.2% | 64.8% | 63.6% | 65.6% | 66.4% | 63.2% | 68.0% | 64.4% | 62.0% | 64.8% | 64.4% | 60.4% | 57.6% | 62.8% | 62.8% | 61.6% | 62.8% | 64.8% | 58.8% | 64.8% | 63.2% | 63.2% | 59.6% | 57.2% | 62.8% | 60.8% | 59.2% | 60.0% | 56.4% | 50.4% | 58.8% | 58.0% | 57.6% | 59.2% | 54.8% | 54.0% | 60.4% | 54.4% | 59.2% | 50.0% | 50.0% | 50.4% | 56.8% | 51.2% | 49.6% | 48.4% | 48.0% | 51.2% | 43.6% | 41.2% | 32.8% |
| Answer evaluationjudge | 94.0% | 92.0% | 93.2% | 91.2% | 91.6% | 90.0% | 90.8% | 90.4% | 90.8% | 92.0% | 89.2% | 91.2% | 86.4% | 86.8% | 82.4% | 86.4% | 86.8% | 86.0% | 87.2% | 82.0% | 88.8% | 87.2% | 89.2% | 84.8% | 88.8% | 74.0% | 83.2% | 85.6% | 85.6% | 85.2% | 87.6% | 83.2% | 84.4% | 76.8% | 80.8% | 79.6% | 72.4% | 81.6% | 80.0% | 70.8% | 67.2% | 69.2% | 68.4% | 83.6% | 70.0% | 75.2% | 70.0% | 71.6% | 69.6% | 66.4% | 75.2% | 68.0% | 68.0% | 71.6% | 69.6% | 70.8% | 61.2% | 66.8% | 60.8% | 62.0% | 63.6% | 68.8% | 67.2% | 59.2% | 61.6% | 57.2% | 72.4% | 57.2% | 62.0% | 57.2% | 68.4% | 69.2% | 55.2% | 70.4% | 62.8% | 64.0% | 54.0% | 56.8% | 57.2% | 55.2% | 60.0% | 52.4% | 44.8% | 57.6% | 50.4% | 45.2% | 50.0% | 44.4% |
| Long documentslong | 76.4% | 78.4% | 80.8% | 70.4% | 69.6% | 69.2% | 66.4% | 74.8% | 73.6% | 74.4% | 70.8% | 56.0% | 76.0% | 73.6% | 77.2% | 69.2% | 70.8% | 66.8% | 67.2% | 68.0% | 68.0% | 72.0% | 66.8% | 71.2% | 66.8% | 65.2% | 66.0% | 69.6% | 70.4% | 66.8% | 62.4% | 60.4% | 66.8% | 65.2% | 65.2% | 64.0% | 68.4% | 62.0% | 61.2% | 66.0% | 60.0% | 62.8% | 66.8% | 35.6% | 63.6% | 55.6% | 63.2% | 46.4% | 56.0% | 64.0% | 50.8% | 61.6% | 63.2% | 56.8% | 54.4% | 54.8% | 59.6% | 55.6% | 59.6% | 57.2% | 57.2% | 8.0% | 54.4% | 58.8% | 46.0% | 58.0% | 52.4% | 50.8% | 50.4% | 46.0% | 43.6% | 55.6% | 57.2% | 7.6% | 51.6% | 6.4% | 48.8% | 55.2% | 46.4% | 6.8% | 50.0% | 46.8% | 46.8% | 0.0% | 34.8% | 38.0% | 38.4% | 26.4% |
| Real documentsrealdoc | 99.2% | 97.2% | 99.2% | 98.8% | 98.8% | 99.2% | 98.4% | 98.0% | 97.6% | 99.2% | 98.8% | 99.2% | 98.4% | 98.0% | 97.2% | 98.4% | 100.0% | 98.8% | 97.6% | 98.0% | 97.2% | 97.2% | 95.2% | 97.2% | 94.8% | 94.8% | 97.6% | 97.6% | 93.6% | 97.2% | 94.8% | 97.6% | 96.0% | 93.6% | 96.4% | 94.4% | 93.6% | 91.6% | 94.0% | 97.6% | 92.0% | 92.0% | 92.8% | 90.8% | 94.0% | 93.2% | 94.0% | 92.8% | 89.2% | 94.8% | 94.4% | 91.6% | 90.8% | 93.6% | 88.8% | 88.8% | 93.2% | 93.6% | 91.2% | 89.2% | 93.6% | 93.2% | 90.8% | 88.4% | 89.6% | 88.4% | 84.8% | 84.8% | 84.8% | 89.2% | 79.6% | 83.2% | 88.8% | 92.0% | 84.8% | 84.0% | 87.6% | 82.4% | 84.0% | 86.4% | 83.6% | 82.8% | 74.8% | 72.8% | 78.0% | 78.8% | 64.4% | 34.8% |
| Search and RAGretrieval | 100.0% | 99.2% | 98.8% | 99.2% | 99.2% | 99.6% | 99.6% | 96.4% | 98.0% | 98.8% | 99.6% | 99.6% | 96.0% | 97.6% | 98.4% | 99.2% | 100.0% | 99.6% | 98.8% | 97.2% | 98.4% | 97.2% | 98.0% | 97.6% | 97.6% | 80.4% | 99.6% | 98.0% | 96.4% | 98.4% | 98.0% | 98.0% | 96.0% | 97.2% | 94.0% | 94.4% | 94.0% | 93.6% | 97.2% | 94.0% | 94.0% | 94.4% | 91.2% | 98.0% | 92.4% | 92.4% | 92.4% | 94.0% | 95.6% | 92.8% | 90.8% | 95.6% | 90.8% | 94.4% | 94.0% | 94.8% | 93.2% | 95.2% | 92.0% | 86.0% | 92.4% | 92.4% | 89.6% | 91.6% | 85.6% | 89.2% | 83.2% | 82.4% | 92.0% | 83.2% | 79.6% | 90.0% | 82.4% | 86.0% | 82.4% | 90.0% | 79.6% | 74.8% | 71.6% | 83.6% | 68.4% | 68.8% | 71.2% | 73.2% | 52.0% | 62.4% | 56.8% | 52.0% |
| Ticket routingrouting | 98.4% | 99.2% | 97.2% | 99.2% | 98.4% | 98.4% | 98.4% | 98.4% | 100.0% | 98.8% | 99.2% | 99.2% | 98.8% | 97.6% | 97.2% | 98.0% | 99.2% | 100.0% | 96.8% | 95.6% | 97.6% | 97.2% | 96.4% | 96.8% | 97.6% | 92.8% | 98.8% | 99.2% | 96.8% | 98.4% | 96.4% | 98.8% | 92.8% | 94.4% | 96.0% | 96.8% | 93.6% | 92.8% | 94.0% | 94.4% | 96.8% | 95.6% | 92.4% | 93.6% | 93.6% | 93.6% | 93.6% | 96.0% | 93.2% | 94.0% | 93.2% | 96.8% | 92.4% | 93.6% | 93.6% | 92.0% | 94.4% | 92.8% | 95.2% | 91.2% | 92.0% | 94.8% | 93.2% | 89.2% | 91.2% | 90.4% | 88.0% | 90.0% | 90.0% | 89.2% | 90.0% | 86.8% | 81.6% | 91.6% | 87.6% | 89.6% | 86.0% | 82.4% | 82.0% | 89.2% | 83.2% | 75.6% | 75.6% | 76.8% | 78.4% | 70.0% | 68.8% | 18.0% |
| Rules and deadlinesrules | 49.2% | 52.4% | 42.0% | 47.2% | 44.0% | 48.0% | 45.6% | 41.6% | 40.8% | 44.0% | 38.4% | 43.6% | 41.6% | 40.0% | 40.0% | 42.4% | 37.2% | 38.8% | 39.2% | 40.8% | 38.4% | 37.2% | 38.4% | 39.2% | 35.2% | 70.4% | 39.2% | 34.0% | 37.6% | 38.8% | 34.0% | 36.8% | 38.4% | 36.8% | 37.6% | 34.8% | 35.2% | 34.4% | 34.8% | 36.8% | 31.6% | 36.0% | 31.6% | 34.8% | 30.8% | 32.4% | 30.4% | 34.4% | 35.6% | 32.8% | 30.8% | 32.0% | 27.2% | 28.4% | 34.0% | 29.2% | 33.6% | 28.8% | 31.2% | 34.0% | 29.2% | 36.8% | 28.4% | 34.4% | 34.4% | 32.4% | 34.8% | 28.0% | 24.8% | 26.4% | 34.0% | 32.4% | 29.2% | 31.6% | 23.6% | 32.0% | 27.2% | 29.6% | 26.8% | 26.4% | 24.4% | 25.2% | 29.2% | 25.2% | 22.4% | 20.8% | 26.0% | 25.2% |
| Scale-based assessmentscore | 85.6% | 84.0% | 85.2% | 81.6% | 83.2% | 83.2% | 84.0% | 82.4% | 83.2% | 80.4% | 82.0% | 81.6% | 79.6% | 77.2% | 78.8% | 78.8% | 79.2% | 75.6% | 76.4% | 78.0% | 77.2% | 77.6% | 77.2% | 77.2% | 80.0% | 78.8% | 74.0% | 76.8% | 77.2% | 74.8% | 74.8% | 77.2% | 74.8% | 71.6% | 73.2% | 73.6% | 71.6% | 69.2% | 66.0% | 68.8% | 68.8% | 65.6% | 72.0% | 69.2% | 66.4% | 67.6% | 66.8% | 71.2% | 68.0% | 67.6% | 68.4% | 63.6% | 62.0% | 64.0% | 68.8% | 68.0% | 64.0% | 67.6% | 66.4% | 67.2% | 65.6% | 70.8% | 62.0% | 59.2% | 67.6% | 59.6% | 56.4% | 58.8% | 60.8% | 62.4% | 63.2% | 53.2% | 56.4% | 64.0% | 60.0% | 62.8% | 55.2% | 49.6% | 57.2% | 60.8% | 54.0% | 44.0% | 49.2% | 51.2% | 44.4% | 42.8% | 41.2% | 20.0% |
| All Polish questions | 85.0% | 85.0% | 84.5% | 83.4% | 83.4% | 82.8% | 82.5% | 82.4% | 82.3% | 82.0% | 81.9% | 81.8% | 81.3% | 80.9% | 80.5% | 80.0% | 80.0% | 79.5% | 79.5% | 79.2% | 79.1% | 79.0% | 79.0% | 78.7% | 78.6% | 78.5% | 77.8% | 77.7% | 77.5% | 77.1% | 76.9% | 76.8% | 76.3% | 74.5% | 74.1% | 73.8% | 73.6% | 72.9% | 72.4% | 72.1% | 70.9% | 70.7% | 70.2% | 70.2% | 70.1% | 70.1% | 70.1% | 69.9% | 69.8% | 69.7% | 69.5% | 69.4% | 69.1% | 68.9% | 68.8% | 68.8% | 68.4% | 68.3% | 68.0% | 67.2% | 66.8% | 66.6% | 66.4% | 66.2% | 66.0% | 63.9% | 62.8% | 62.5% | 62.4% | 62.2% | 62.2% | 61.9% | 61.7% | 61.6% | 61.4% | 60.7% | 59.2% | 58.0% | 57.8% | 57.5% | 57.1% | 53.7% | 52.7% | 49.2% | 48.7% | 48.2% | 45.9% | 29.4% |
With around 250 questions per category, differences below roughly 6 percentage points may be noise. This is a rule of thumb, not a significance test for every pair. Source: Polish categories from the Werdykt export dated 11 October 2026. Qwen3-4B-Instruct-2507, Phi-4-mini-instruct and Qwen3-Reranker-4B are controls.
abstain
Insufficient information
Choose an answer or determine that the available information is insufficient.
action
Agent’s next step
An action consistent with the stated policy and process state.
contract
Contracts
Interpret clauses and make decisions based on contract terms.
judge
Answer evaluation
Assess answer correctness and compare two proposed answers.
long
Long documents
Make decisions based on documents containing 3,000–10,000 tokens.
realdoc
Real documents
Questions about the content of authentic official documents.
retrieval
Search and RAG
Passage relevance, context sufficiency and whether a source supports a claim.
routing
Ticket routing
Choose a queue, intent and priority according to the stated rules.
rules
Rules and deadlines
Apply rules, deadlines and amount-based conditions.
score
Scale-based assessment
Choose a level consistent with the scale description.
04 / FROM BASAL-1.0 TO BASAL-2.0
Progress on Polish tasks.
basal-1.0 → basal-1.5
Historical table from the basal-1.5 release report (3 October 2026). Three separate Polish tests, independent of the Werdykt table. The strongest signal: 93.1% vs 87.4% on 3,000 new decisions created after the weights were frozen.
| Test | basal-1.0 · 4.5B | basal-1.5 · 4.5B | basal-1.5-max · 11B | basal-1.0 · 1.5B | basal-1.5-mini · 1.5B |
|---|---|---|---|---|---|
| Hidden Polish test3,000 new decisions; created after the weights were frozen | 87.4% | 93.1% | 93.3% | 88.9% | 91.0% |
| Polish decisions7,081 cases | 88.6% | 92.3% | 94.0% | 85.1% | 87.7% |
| General knowledge in Polish3,079 cases | 73.7% | 74.1% | 79.4% | 65.6% | 66.7% |
HOW TO READ THESE RESULTS
Decision specialisation, with measurable progress.
The main model gains 5.7 percentage points on the hidden Polish test (95% difference interval: +4.5 to +6.8 pp). Max scores similarly on this test and leads in general knowledge. General-knowledge improvements for the main model and mini are not statistically significant.
basal-1.5 → basal-2.0
In Werdykt, basal-2.0 gains 0.7–1.2 percentage points, within the confidence intervals, while calibration error (ECE) falls: mini 0.086 → 0.035, 4.5B 0.068 → 0.060, max 0.074 → 0.044 (basal-1.5 as in the main table: 4.5B and max through SGLang, mini in fast mode). Below are six separate tests. Both generations used the same basal v2.0.0 engine on the same questions, giving paired differences with 95% bootstrap intervals. We mark improvements or declines only when the interval excludes zero.
| Test | Size | basal-1.5 | basal-2.0 | Difference [95%] |
|---|---|---|---|---|
| PL-Holdouthidden Polish test; 4,000 decisions, with task families excluded from training | mini · 1.5B | 83.7% | 82.9% | −0.8 pp [−1.6; 0.0] · no significant change |
| 4.5B | 85.0% | 85.6% | +0.5 pp [−0.2; +1.3] · no significant change | |
| max · 11B | 86.8% | 86.2% | −0.6 pp [−1.6; +0.4] · no significant change | |
| Polish decisions7,081 cases | mini · 1.5B | 88.0% | 86.7% | −1.4 pp [−2.0; −0.8] · spadek |
| 4.5B | 92.2% | 92.4% | +0.1 pp [−0.4; +0.7] · no significant change | |
| max · 11B | 94.0% | 93.8% | −0.2 pp [−0.8; +0.4] · no significant change | |
| General knowledge in Polish3,079 cases | mini · 1.5B | 67.1% | 69.3% | +2.2 pp [+1.0; +3.4] · poprawa |
| 4.5B | 74.6% | 76.2% | +1.6 pp [+0.4; +2.8] · poprawa | |
| max · 11B | 79.7% | 81.1% | +1.4 pp [+0.3; +2.5] · poprawa | |
| Real Polish documents158 questions about documents outside the training data | mini · 1.5B | 91.8% | 91.8% | 0.0 pp [−3.8; +3.8] · no significant change |
| 4.5B | 94.9% | 94.9% | 0.0 pp [−3.2; +3.2] · no significant change | |
| max · 11B | 96.2% | 97.5% | +1.3 pp [−1.3; +3.8] · no significant change | |
| English test set1,479 cases | mini · 1.5B | 59.4% | 60.5% | +1.1 pp [−0.7; +2.9] · no significant change |
| 4.5B | 64.3% | 64.4% | +0.1 pp [−1.7; +1.9] · no significant change | |
| max · 11B | 66.1% | 67.3% | +1.1 pp [−0.6; +3.1] · no significant change | |
| English panel10 tasks; difference and interval calculated at question level | mini · 1.5B | 58.2% | 56.1% | −1.5 pp [−2.9; 0.0] · spadek |
| 4.5B | 68.8% | 65.2% | −3.6 pp [−5.3; −2.0] · spadek | |
| max · 11B | 73.7% | 71.9% | −2.5 pp [−4.0; −1.0] · spadek |
HOW TO READ THESE RESULTS
Better calibration and general knowledge. Not every test improves.
All three sizes improve in Polish general knowledge (+1.4 to +2.2 pp). All decline on the English panel (−1.5 to −3.6 pp), and mini loses 1.4 pp on Polish decisions. Other differences fall within the uncertainty intervals. Test the model on your own data before upgrading a live workflow.
05 / DECISION CONFIDENCE · 11 Oct 2026
Download analysis JSON
How many decisions can you automate?
The model selects an answer and assigns it a probability. You set a threshold: accept higher-confidence decisions automatically and send the rest to a person. We measure how much work passes this threshold — and the error rate among accepted decisions.
- 01 The model selects an answer
- 02 You check its confidence
- 03 Automation or human review
of decisions accepted automatically
basal-omni-1.0-max · measured error: 5.02%
model confidence vs test performance
basal-omni-1.0-max is overconfident on average. Validate the threshold on data from your own process.
A target is not a guarantee. The threshold was selected on one half of the questions; automation and error were measured on the other half. At a 1% target, basal-2.0-max had a measured error of 1.34%, and basal-omni-1.0-max 1.45%. The analysis covers 86 systems from the current Werdykt export dated 11 October 2026, including three basal-2.0 and two basal-omni models. These are 5,000 PL+EN text questions. The results do not measure image or audio calibration.
For the main basal-2.0 model, the threshold selected for a 5% target accepts 51.3% of decisions, with a measured error of 5.08%. The charts show the entire new family and Jev by default; you can add other systems below.
Choose systems for the charts
Click a name to show or hide its series. The selection applies to both charts.
No system selected. Expand “Choose systems for the charts” or restore the default selection.
Can you trust the confidence?
Reported confidence and actual accuracy in 10 bins. Points below the diagonal indicate overconfidence. Larger points represent more questions.
Scroll horizontally to see the full scale.
How much work should the model handle?
Lines: accuracy as decisions are accepted from highest confidence down, across the whole set. Markers: a separate measurement on the test half, after selecting a threshold for a 1% (○) or 5% (□) target.
Scroll horizontally to see the full scale.
Point to a marker to read its value. Keyboard: Tab selects a series, ← → move between points. All systems and threshold results are also in the table.
Interactive charts require JavaScript and downloaded data. Full results are available in the table below and in JSON.
Comparison 86 systems
Target columns show automatically accepted decisions, measured error and the selected threshold. Lower ECE, Brier and AURC are better.
86 systems · sorted by automation at the 5% target by default · scroll horizontally.
| System / parameters | Accuracy ↑ | Mean confidence | ECE ↓ | Brier ↓ | AURC ↓ | Error target: 1% Automation ↑ | Error target: 5% Automation ↑ |
|---|---|---|---|---|---|---|---|
| René-1 31B FP831B · open weightsModel server | 85.36% | 86.95% | 0.017 | 0.195 | 0.025 | 65.77%Error: 1.38%Threshold: 0.9300 | 81.52%Error: 5.23%Threshold: 0.6900 |
| pplx-decider-v1.1-27b (Decisions API)27B · APIAPI | 85.18% | 90.80% | 0.057 | 0.207 | 0.024 | 61.19%Error: 1.03%Threshold: 0.9922 | 81.25%Error: 4.47%Threshold: 0.8117 |
| Mercury Decide (OpenRouter Decisions API)Not disclosed · APIAPI | 84.48% | 92.94% | 0.085 | 0.233 | 0.029 | 57.95%Error: 1.23%Threshold: 0.9979 | 80.06%Error: 4.93%Threshold: 0.9442 |
| pplx-decider-v1-27b27B · open weightsModel server | 81.80% | 80.02% | 0.043 | 0.248 | 0.030 | 60.56%Error: 1.30%Threshold: 0.8605 | 78.80%Error: 4.86%Threshold: 0.5460 |
| decisio v0.8.0 on gemma-4-31B-it31B · open weightsModel server | 83.48% | 89.78% | 0.063 | 0.237 | 0.035 | 9.63%Error: 0.82%Threshold: 0.9955 | 76.94%Error: 4.46%Threshold: 0.8787 |
| deck-31B31B · open weightsModel server | 83.54% | 94.11% | 0.106 | 0.259 | 0.035 | 45.08%Error: 1.40%Threshold: 0.9966 | 76.63%Error: 4.33%Threshold: 0.9656 |
| Jebadiah 27B27B · open weightsModel server | 81.96% | 73.60% | 0.093 | 0.242 | 0.033 | 52.23%Error: 1.13%Threshold: 0.7978 | 76.35%Error: 4.40%Threshold: 0.5573 |
| basal-omni-1.0-max12B · open weightsbasal-serve · eager | 82.28% | 87.71% | 0.054 | 0.241 | 0.036 | 60.09%Error: 1.45%Threshold: 0.9665 | 75.52%Error: 5.02%Threshold: 0.8161 |
| Surogate Rune 26B-A4B v326B-A4B · open weightssurogate · C++ | 81.68% | 82.09% | 0.024 | 0.240 | 0.038 | 50.93%Error: 1.40%Threshold: 0.9266 | 73.31%Error: 5.12%Threshold: 0.7146 |
| Jev 1.13.0— · APIAPI | 81.64% | 82.92% | 0.016 | 0.235 | 0.036 | 49.39%Error: 1.04%Threshold: 0.9700 | 72.84%Error: 4.07%Threshold: 0.7200 |
| Eikos-27B-FP8 (v1.3)27B · open weightsModel server | 82.12% | 84.11% | 0.060 | 0.246 | 0.037 | 51.72%Error: 1.30%Threshold: 0.9132 | 72.40%Error: 4.80%Threshold: 0.7902 |
| Decider chat · Gemma-4-31B31B · open weightsModel server | 82.46% | 75.83% | 0.093 | 0.248 | 0.041 | 15.16%Error: 0.52%Threshold: 0.9271 | 71.97%Error: 4.22%Threshold: 0.7118 |
| Bobcat Flash 1.226B-A4B · open weightsModel server | 80.22% | 88.02% | 0.078 | 0.273 | 0.042 | 46.90%Error: 0.93%Threshold: 0.9930 | 71.46%Error: 5.41%Threshold: 0.8676 |
| simple-jev · Qwen3.8-27B (featherless)27B · open weightsModel server | 80.14% | 84.38% | 0.043 | 0.261 | 0.041 | 49.74%Error: 1.03%Threshold: 0.9825 | 70.04%Error: 4.34%Threshold: 0.7982 |
| Quyet-1.0-Large31.3B (30.7B text) · open weightsModel server | 83.28% | 92.52% | 0.088 | 0.263 | 0.044 | 26.10%Error: 2.12%Threshold: 0.9973 | 69.96%Error: 4.74%Threshold: 0.9536 |
| Winnow-12B (Q8)12B · open weightsModel server | 78.50% | 86.99% | 0.085 | 0.288 | 0.047 | 47.37%Error: 1.50%Threshold: 0.9857 | 69.80%Error: 5.60%Threshold: 0.9279 |
| Clef (OpenRouter Decisions API)27B · APIAPI | 80.76% | 85.42% | 0.047 | 0.262 | 0.045 | 19.23%Error: 0.21%Threshold: 0.9843 | 68.30%Error: 3.99%Threshold: 0.8580 |
| Liquid d1— · APIAPI | 80.78% | 86.01% | 0.054 | 0.262 | 0.044 | 29.41%Error: 0.40%Threshold: 0.9973 | 67.82%Error: 3.84%Threshold: 0.8797 |
| Xor 26B-A4B 1.026B-A4B · open weightsModel server | 79.76% | 72.29% | 0.086 | 0.275 | 0.044 | 44.77%Error: 0.53%Threshold: 0.8152 | 67.67%Error: 3.85%Threshold: 0.6416 |
| Reflex 27B (Decision Index 0.2 build)27B · open weightsModel server | 79.10% | 78.40% | 0.030 | 0.273 | 0.050 | 35.93%Error: 1.10%Threshold: 0.9568 | 67.39%Error: 4.86%Threshold: 0.7220 |
| decider-12b v212B · open weightsModel server | 78.90% | 76.99% | 0.039 | 0.270 | 0.045 | 51.60%Error: 1.45%Threshold: 0.8543 | 66.88%Error: 4.13%Threshold: 0.7068 |
| Cygnet12B · open weightsModel server | 78.52% | 81.42% | 0.045 | 0.283 | 0.049 | 45.83%Error: 1.21%Threshold: 0.9330 | 66.25%Error: 4.59%Threshold: 0.8015 |
| basal-1.5-max11B · open weightsSGLang | 77.28% | 84.68% | 0.074 | 0.307 | 0.054 | 42.52%Error: 0.93%Threshold: 0.9861 | 65.61%Error: 4.99%Threshold: 0.8677 |
| basal-2.0-max11B · open weightsbasal v2.0.0 · fast | 78.40% | 82.57% | 0.044 | 0.286 | 0.053 | 35.33%Error: 1.34%Threshold: 0.9714 | 65.61%Error: 4.51%Threshold: 0.8146 |
| GPT-6 Luna Decisions— · APIAPI | 78.38% | 80.75% | 0.025 | 0.285 | 0.055 | 0.00%Error: —Threshold: — | 65.38%Error: 5.62%Threshold: 0.7800 |
| Microsoft-Decision-1 (OpenRouter Decisions API)Not disclosed · APIAPI | 80.02% | 88.65% | 0.087 | 0.285 | 0.051 | 27.71%Error: 0.85%Threshold: 0.9983 | 64.31%Error: 4.05%Threshold: 0.9345 |
| basal-omni-1.0-mini4B · open weightsbasal-serve · eager | 76.88% | 86.37% | 0.095 | 0.319 | 0.059 | 39.01%Error: 1.52%Threshold: 0.9900 | 63.76%Error: 4.46%Threshold: 0.9016 |
| decider-12b v112B · open weightsModel server | 77.10% | 75.86% | 0.062 | 0.296 | 0.054 | 47.14%Error: 1.42%Threshold: 0.8607 | 63.60%Error: 4.28%Threshold: 0.7391 |
| JEVfire (Qwen3.8-27B-FP8)27B · open weightsModel server | 77.74% | 79.58% | 0.024 | 0.292 | 0.057 | 35.10%Error: 1.57%Threshold: 0.9861 | 62.10%Error: 4.51%Threshold: 0.8176 |
| Xor 1.235B-A3B · open weightsModel server | 77.22% | 78.86% | 0.029 | 0.299 | 0.058 | 27.95%Error: 0.71%Threshold: 0.9788 | 61.82%Error: 4.21%Threshold: 0.7695 |
| Jev-Omni12B · open weightsModel server | 75.34% | 84.03% | 0.087 | 0.333 | 0.062 | 37.66%Error: 0.52%Threshold: 0.9936 | 60.76%Error: 4.16%Threshold: 0.9234 |
| H2O-Lightning-4B v1.14B · open weightsModel server | 75.30% | 76.97% | 0.041 | 0.319 | 0.068 | 26.41%Error: 0.75%Threshold: 0.9479 | 57.17%Error: 4.01%Threshold: 0.8326 |
| Decider chat · Qwen3.6-27B27B · open weightsModel server | 76.86% | 73.16% | 0.052 | 0.307 | 0.063 | 25.90%Error: 0.76%Threshold: 0.9109 | 56.81%Error: 4.17%Threshold: 0.7596 |
| JPT-9B9B · open weightsModel server | 75.18% | 80.08% | 0.049 | 0.325 | 0.067 | 25.78%Error: 0.46%Threshold: 0.9761 | 55.39%Error: 3.71%Threshold: 0.8787 |
| djev26B (4B active) · open weightsModel server | 69.80% | 84.34% | 0.105 | 0.448 | 0.087 | 9.83%Error: 2.01%Threshold: 0.9994 | 55.19%Error: 5.72%Threshold: 0.9263 |
| Solar Decide— · APIAPI | 81.32% | 92.32% | 0.110 | 0.296 | 0.064 | 0.00%Error: —Threshold: — | 54.52%Error: 4.56%Threshold: 0.9806 |
| sonda-1.0-4B4B · open weightsvLLM | 70.72% | 80.79% | 0.084 | 0.401 | 0.084 | 32.18%Error: 1.72%Threshold: 0.9904 | 52.74%Error: 4.34%Threshold: 0.9014 |
| basal-2.04.5B · open weightsbasal v2.0.0 · fast | 72.86% | 78.76% | 0.060 | 0.362 | 0.086 | 11.92%Error: 0.66%Threshold: 0.9872 | 51.28%Error: 5.08%Threshold: 0.8761 |
| Imajev-4B4B · open weightsModel server | 68.16% | 76.60% | 0.039 | 0.476 | 0.097 | 6.24%Error: 0.00%Threshold: 0.9931 | 50.41%Error: 4.62%Threshold: 0.8078 |
| basal-1.54.5B · open weightsSGLang | 72.12% | 78.91% | 0.068 | 0.371 | 0.088 | 22.46%Error: 1.23%Threshold: 0.9926 | 50.14%Error: 5.04%Threshold: 0.9164 |
| Decision 1.0 Lux 9B9B · open weightsModel server | 74.94% | 77.00% | 0.022 | 0.331 | 0.077 | 13.26%Error: 0.60%Threshold: 0.9828 | 48.68%Error: 4.62%Threshold: 0.8681 |
| Hopper (G) 1.24B · open weightsModel server | 71.10% | 75.58% | 0.070 | 0.374 | 0.089 | 23.96%Error: 1.15%Threshold: 0.9379 | 48.56%Error: 3.82%Threshold: 0.8699 |
| MedDecider-9B9.7B · open weightsModel server | 73.28% | 71.52% | 0.025 | 0.341 | 0.080 | 29.21%Error: 0.81%Threshold: 0.9268 | 47.81%Error: 4.54%Threshold: 0.8154 |
| decider-4b v24B · open weightsModel server | 70.00% | 73.31% | 0.038 | 0.375 | 0.095 | 25.23%Error: 1.10%Threshold: 0.9565 | 46.55%Error: 4.24%Threshold: 0.8476 |
| JPT-4B4B · open weightsModel server | 71.14% | 78.36% | 0.072 | 0.376 | 0.093 | 2.01%Error: 0.00%Threshold: 0.9968 | 44.61%Error: 3.45%Threshold: 0.9254 |
| DREX v1.5 (OpenRouter Decisions API)Not disclosed · APIAPI | 74.88% | 86.19% | 0.113 | 0.363 | 0.084 | 0.00%Error: —Threshold: — | 44.49%Error: 4.17%Threshold: 0.9838 |
| Decision-2.0-Nox-4B4B (4.21B) · open weightseager · H100 NVL | 71.64% | 68.89% | 0.032 | 0.362 | 0.092 | 21.95%Error: 0.54%Threshold: 0.9303 | 44.14%Error: 4.20%Threshold: 0.8079 |
| JevK5 v0.34B · open weightsModel server | 68.86% | 76.41% | 0.075 | 0.404 | 0.107 | 13.50%Error: 1.75%Threshold: 0.9914 | 41.49%Error: 4.38%Threshold: 0.9029 |
| Clef Flash (OpenRouter Decisions API)9B · APIAPI | 71.60% | 77.88% | 0.063 | 0.383 | 0.099 | 13.26%Error: 1.19%Threshold: 0.9639 | 39.28%Error: 4.22%Threshold: 0.9212 |
| MedDecider-4B4.7B · open weightsModel server | 68.90% | 69.73% | 0.020 | 0.385 | 0.104 | 17.17%Error: 0.23%Threshold: 0.9639 | 39.28%Error: 3.82%Threshold: 0.8562 |
| spark-s1-4b-v64B · open weightsopen_spark_jev · HF | 69.38% | 85.34% | 0.160 | 0.453 | 0.107 | 24.52%Error: 1.45%Threshold: 0.9978 | 39.24%Error: 3.62%Threshold: 0.9921 |
| Clef-Flash9B · open weightsModel server | 71.58% | 77.87% | 0.063 | 0.383 | 0.099 | 10.74%Error: 0.37%Threshold: 0.9675 | 39.01%Error: 4.25%Threshold: 0.9222 |
| jev-local (Qwen3.5-9B, HF scorer)9B · open weightsjevlocal · HF scorer | 62.18% | 59.15% | 0.090 | 0.550 | 0.132 | 20.69%Error: 1.15%Threshold: 0.8063 | 38.45%Error: 5.03%Threshold: 0.6658 |
| Tev1 4B Experimental4B · APIAPI | 68.62% | 78.62% | 0.100 | 0.410 | 0.111 | 13.46%Error: 0.29%Threshold: 0.9969 | 38.10%Error: 5.18%Threshold: 0.9599 |
| JevK5 v0.2 (4B)4B · open weightsjevk5-serve | 66.72% | 72.92% | 0.062 | 0.416 | 0.120 | 10.07%Error: 0.39%Threshold: 0.9883 | 37.47%Error: 4.85%Threshold: 0.9004 |
| Kev 4B (OpenRouter Decisions API)4B · APIAPI | 67.44% | 65.75% | 0.033 | 0.426 | 0.115 | 21.08%Error: 1.50%Threshold: 0.9119 | 37.19%Error: 4.03%Threshold: 0.7992 |
| Kev 1.0 (kev-4b)4B · open weightskev.serve | 68.36% | 65.35% | 0.034 | 0.397 | 0.114 | 21.67%Error: 1.46%Threshold: 0.9100 | 36.72%Error: 4.09%Threshold: 0.8048 |
| Qwen3-32B, zero-shot (jqv)32B · open weightsModel server | 70.54% | 71.64% | 0.039 | 0.385 | 0.103 | 17.33%Error: 1.37%Threshold: 0.9340 | 36.00%Error: 4.28%Threshold: 0.8569 |
| Decision 4B v1.24B · open weightsModel server | 70.16% | 70.05% | 0.024 | 0.386 | 0.103 | 9.51%Error: 0.41%Threshold: 0.9738 | 35.61%Error: 3.66%Threshold: 0.8698 |
| Jet v6.24B · open weightsModel server | 67.34% | 82.17% | 0.148 | 0.472 | 0.127 | 15.44%Error: 1.79%Threshold: 0.9640 | 35.37%Error: 5.92%Threshold: 0.9384 |
| Decision 4B v1.14B · open weightsModel server | 69.02% | 71.49% | 0.025 | 0.397 | 0.110 | 13.11%Error: 1.51%Threshold: 0.9724 | 33.99%Error: 3.37%Threshold: 0.9043 |
| Plumb-4B4B · open weightsModel server | 69.04% | 78.96% | 0.099 | 0.423 | 0.116 | 1.74%Error: 0.00%Threshold: 0.9969 | 33.87%Error: 3.73%Threshold: 0.9339 |
| Malkuth-4B4B · open weightsModel server | 63.36% | 69.92% | 0.052 | 0.474 | 0.143 | 16.27%Error: 1.46%Threshold: 0.9664 | 32.49%Error: 5.59%Threshold: 0.8939 |
| basal-1.0 · 4.5B4.5B · open weightsbasal · fast | 66.04% | 72.74% | 0.070 | 0.446 | 0.139 | 17.02%Error: 0.46%Threshold: 0.9773 | 31.78%Error: 4.72%Threshold: 0.9098 |
| Hopper4B · open weightsModel server | 70.28% | 70.12% | 0.059 | 0.395 | 0.111 | 7.30%Error: 1.08%Threshold: 0.9947 | 30.40%Error: 3.77%Threshold: 0.9291 |
| decider-35b-a3b NVFP435B-A3B · open weightsModel server | 70.38% | 74.35% | 0.040 | 0.385 | 0.109 | 5.96%Error: 1.32%Threshold: 0.9973 | 30.20%Error: 4.58%Threshold: 0.9369 |
| sonda-1.1-4B4B · open weightsModel server | 64.02% | 76.14% | 0.121 | 0.468 | 0.142 | 14.45%Error: 1.91%Threshold: 0.9869 | 30.16%Error: 5.24%Threshold: 0.9544 |
| Clef Omni (OpenRouter Decisions API)Not disclosed · APIAPI | 70.14% | 77.36% | 0.072 | 0.403 | 0.115 | 3.71%Error: 1.06%Threshold: 0.9744 | 26.02%Error: 4.86%Threshold: 0.9208 |
| basal-2.0-mini1.5B · open weightsbasal v2.0.0 · fast | 61.84% | 64.25% | 0.035 | 0.477 | 0.165 | 15.16%Error: 1.82%Threshold: 0.9138 | 25.42%Error: 5.12%Threshold: 0.8566 |
| SemIf4B · open weightsModel server | 58.80% | 73.51% | 0.091 | 0.588 | 0.166 | 4.97%Error: 0.79%Threshold: 0.9989 | 24.83%Error: 5.56%Threshold: 0.9715 |
| local-jev (Qwen3.5-4B)4B · open weightslocal-jev · HF | 62.98% | 64.27% | 0.020 | 0.458 | 0.158 | 4.30%Error: 0.92%Threshold: 0.9854 | 24.71%Error: 6.39%Threshold: 0.8895 |
| typecastlm v1.3.0 (Qwen3.5 3.8B)3.8B · open weightstypecastlm · local | 62.88% | 62.03% | 0.067 | 0.459 | 0.151 | 10.34%Error: 1.15%Threshold: 0.8927 | 23.81%Error: 5.14%Threshold: 0.8223 |
| metask-jev-4b4B · open weightsModel server | 61.16% | 73.71% | 0.066 | 0.572 | 0.151 | 9.59%Error: 1.65%Threshold: 0.9510 | 23.73%Error: 4.99%Threshold: 0.9196 |
| open-alternative-jev (so1, Qwen3.5-4B)4B · open weightsso1 · HF + HTTP adapter | 60.24% | 73.66% | 0.134 | 0.528 | 0.182 | 7.86%Error: 1.01%Threshold: 0.9937 | 22.11%Error: 5.18%Threshold: 0.9677 |
| basal-1.5-mini1.5B · open weightsbasal · fast | 60.66% | 69.26% | 0.086 | 0.498 | 0.174 | 11.92%Error: 1.66%Threshold: 0.9846 | 21.87%Error: 4.33%Threshold: 0.9554 |
| Manchego v2.14B · open weightsModel server | 63.04% | 74.95% | 0.119 | 0.502 | 0.178 | 3.91%Error: 1.01%Threshold: 0.9978 | 19.98%Error: 3.95%Threshold: 0.9577 |
| decider-2b (v11)2B · open weightsdecider-ai 1.8.1 | 58.66% | 70.99% | 0.123 | 0.538 | 0.204 | 9.79%Error: 3.23%Threshold: 0.9881 | 17.02%Error: 5.57%Threshold: 0.9649 |
| Solar Decide Flash— · APIAPI | 60.16% | 75.24% | 0.151 | 0.534 | 0.186 | 5.21%Error: 0.00%Threshold: 0.9968 | 16.66%Error: 2.84%Threshold: 0.9842 |
| Decision 2B (preview)2B · open weightsmodel.decide + HTTP adapter | 54.62% | 62.45% | 0.030 | 0.627 | 0.215 | 4.66%Error: 0.85%Threshold: 0.9574 | 16.27%Error: 6.80%Threshold: 0.8789 |
| lev4B · open weightsModel server | 62.86% | 73.91% | 0.111 | 0.484 | 0.168 | 0.00%Error: —Threshold: — | 11.05%Error: 3.93%Threshold: 0.9746 |
| Nemotron Diffusion 8B (Nemotron_Jev)8B · open weightsNemotron_Jev · vLLM | 51.54% | 66.62% | 0.151 | 0.620 | 0.276 | 4.66%Error: 2.54%Threshold: 0.9931 | 10.03%Error: 7.87%Threshold: 0.9772 |
| basal-1.0 · 1.5B1.5B · open weightsbasal · fast | 52.24% | 63.65% | 0.114 | 0.601 | 0.289 | 0.00%Error: —Threshold: — | 5.53%Error: 6.43%Threshold: 0.9820 |
| Malkuth-2B2B · open weightskev.serve | 49.98% | 64.45% | 0.134 | 0.644 | 0.302 | 0.00%Error: —Threshold: — | 2.80%Error: 2.82%Threshold: 0.9909 |
| Certo v10.4B · open weightscerto · HTTP adapter | 28.26% | 29.71% | 0.017 | 0.730 | 0.616 | 0.00%Error: —Threshold: — | 0.00%Error: —Threshold: — |
| JEV-Qwen3.5-9B-Base-NVFP49B · open weightsSGLang · NVFP4 + HTTP adapter | 57.26% | 63.32% | 0.061 | 0.539 | 0.235 | 0.00%Error: —Threshold: — | 0.00%Error: —Threshold: — |
| Qwen3-Reranker-4B (as a decider)4B · open weightsTransformers · reranker adapter | 47.20% | 33.67% | 0.137 | 0.669 | 0.414 | 0.00%Error: —Threshold: — | 0.00%Error: —Threshold: — |
How to read the 1% and 5% targets. We select thresholds on one half of the questions, using a fixed split based on identifier hashes, then apply them to the other half. “Automation” is the proportion accepted on that second half; “Error” is the error rate among accepted answers. The result can exceed the target. “—” means no threshold was found or no decisions were accepted for which error could be measured.
Confidence and comparison scope. Confidence is the highest answer probability returned by a system. The analysis covers 86 systems with calibration data in the current Werdykt export. We omit runs without such data, including GPT, Gemini and Claude, which returned a selected answer without a comparable probability distribution under this protocol.
ECE, Brier i AURC. ECE is the weighted mean absolute difference between confidence and accuracy across 10 equal-width bins. Brier measures error across the full answer probability distribution (multiclass version). AURC is the area under the risk–coverage curve: it measures how well confidence ranks answers from most to least reliable.
Curves vs threshold measurements. Calibration and accuracy curves describe the entire set. In the automation chart, 1% and 5% markers show actual coverage and accuracy on the separate test half; they need not lie on the whole-set curve. Failed requests are never accepted automatically and count as errors in overall test accuracy. Calibration bins include responses with available probabilities.
basal conditions. Results use the same runs as the main ranking: basal-2.0 uses the fast engine, basal-omni uses eager; BF16 and calibration enabled, answers in --deterministic mode. Historical models remain reference points, with their engine listed by name. Thresholds estimated here are an analysis of Werdykt, not ready-made max_error settings for any workflow. We publish only aggregates, without test questions or documents.
MEASUREMENT CONDITIONS
What exactly does Werdykt measure?
The score measures selection from the provided options. It does not rate an entire product or guarantee correctness in a new workflow.
The same problem, one call
Every system receives the same state, question and options. Chat models receive a shared fixed prompt requesting JSON; decision servers receive equivalent System One fields. An unparsed answer or failed call counts as an error.
Accuracy and uncertainty
The main score is mean accuracy across 20 groups: 10 categories × Polish and English, with 250 questions each. The 95% intervals use stratified bootstrap: 1,000 resamples within each group. The table's interval refers only to the PL + EN score, even after changing the filter.
Time, hardware and cost
Median and p95: 200 shared questions, individual requests without a queue. Most open models run on one H100 80 GB, Nox on H100 NVL 94 GB, and several marked systems on RTX PRO 6000 Blackwell 96 GB. The client runs next to the server. We calculate H100 and H100 NVL costs at USD 3.95/h, Blackwell at USD 2.09/h. API latency includes the provider network round trip. Local-model throughput is measured with 16 concurrent requests. API models retain the reasoning setting listed in the table; Jev uses its own API.
basal always uses calibration. basal-1.0 and basal-1.5 evaluate both option orders; basal-2.0 evaluates one, the default for its request format. In this table, basal-1.5 4.5B and max run through SGLang; their fast-mode medians were 31.2 and 44.7 ms respectively. basal-2.0 runs on the basal v2.0.0 engine in fast mode.
Hidden set and public examples
We do not publish hidden questions. No Werdykt question, document or excerpt appears in basal's training data. The examples below are separate synthetic illustrations written for this page. They contain no excerpts from source documents and were not used to calculate the ranking. The next benchmark version is planned to replace about 30% of each group and report results on the retained portion.
What do these decisions look like?
Three fictional scenarios per category, in Polish and English. We wrote them from scratch for this page, without copying source documents. Role labels such as “Customer A” replace people and organisations. These examples illustrate tasks and were not used to calculate the ranking. All examples are shown in English. PL or EN indicates the language of the original illustration.
Insufficient information 3 examples
1 · original PL · How many units of product B can be allocated to a new order?
Synthetic example · text written for this page.
Fictional warehouse register. Product A: 18 units in stock, 7 reserved. Product B: 9 units in stock. The register does not state how many units of product B are reserved. Stock available for a new order equals stock on hand minus reservations.
Possible answers
- 9 units
- 2 units
- The register does not provide enough information
Reference answer: The register does not provide enough information
2 · original EN · Who is assigned to the afternoon shift?
Synthetic example · text written for this page.
Synthetic handover log. Operator A covers the morning shift. Operator B has requested an exchange with Operator C for the afternoon shift. A request becomes effective only after approval by the shift lead. The log includes no approval and does not identify the original afternoon operator.
Possible answers
- Operator B
- Operator C
- The log does not establish the assignment
Reference answer: The log does not establish the assignment
3 · original PL · Which operating time limit applies according to these documents?
Synthetic example · text written for this page.
Demonstration device specification. The draft document states an operating limit of 8 hours. The approved version states a limit of 6 hours. The process instructions say that the approved version supersedes the draft.
Possible answers
- 6 hours
- 8 hours
- Cannot be determined
Reference answer: 6 hours
Agent’s next step 3 examples
1 · original PL · What should the agent do next?
Synthetic example · text written for this page.
Simulated ticket handling. Procedure: first check completeness, then verify eligibility, and finally propose a solution. Process state: completeness confirmed; eligibility not yet checked. Available tools: check completeness, verify eligibility, prepare a proposal, close the ticket.
Possible answers
- Verify eligibility
- Prepare a proposal
- Close the ticket
Reference answer: Verify eligibility
2 · original EN · Which action is allowed next?
Synthetic example · text written for this page.
Synthetic deployment exercise. Policy: a release may reach the staging environment only after both the build and automated checks pass. Production additionally requires an explicit approval. Current state: build passed, checks passed, staging not started, production approval absent.
Possible answers
- Deploy to staging
- Deploy directly to production
- Skip the remaining steps
Reference answer: Deploy to staging
3 · original PL · What should the agent do now?
Synthetic example · text written for this page.
A fictional service call returned code 429 and the header Retry-After: 20. The procedure allows no more than three attempts and requires waiting for the time specified in the header. The first attempt has completed; retrying the operation is safe.
Possible answers
- Retry immediately
- Wait 20 seconds and retry
- Treat the operation as successful
Reference answer: Wait 20 seconds and retry
Contracts 3 examples
1 · original PL · What reduction do these terms provide?
Synthetic example · text written for this page.
Demonstration contract: the monthly service fee is 200 billing points. If monthly availability falls below 99%, a reduction of 10% of that fee applies. Availability in the month under review was 98.5%. No other reductions apply.
Possible answers
- 0 points
- 20 points
- 100 points
Reference answer: 20 points
2 · original EN · Is the proposed task permitted by the stated agreement?
Synthetic example · text written for this page.
Fictional service agreement. The supplier may use Subcontractor A for storage only. Access to message content requires separate written permission. No such permission has been granted. The proposed task is to let Subcontractor A read messages to prepare summaries.
Possible answers
- Yes, storage permission includes reading
- No, separate permission is required
- Yes, if the summaries are short
Reference answer: No, separate permission is required
3 · original PL · What extra charge follows from this contract?
Synthetic example · text written for this page.
Fictional rental contract: the base package covers 4 days. Each additional day, including a partial day, costs 15 points. The recipient kept the equipment for 6 full days. There is no grace period or other surcharge.
Possible answers
- 15 points
- 30 points
- 90 points
Reference answer: 30 points
Answer evaluation 3 examples
1 · original EN · How should the candidate answer be assessed?
Synthetic example · text written for this page.
Synthetic reference: Plan A permits three devices and includes email support. It does not include telephone support. User question: how many devices are allowed and is telephone support included? Candidate answer: three devices are allowed and telephone support is included.
Possible answers
- Fully correct
- Partly correct: the support claim is wrong
- Unsupported in every part
Reference answer: Partly correct: the support claim is wrong
2 · original PL · Which answer covers all required actions?
Synthetic example · text written for this page.
Fictional instructions: when moving a project, keep its name, specify the new directory and create a backup. The question asks for all required actions. Answer A lists all three actions. Answer B lists keeping the name and specifying the directory, but omits the backup.
Possible answers
- Answer A
- Answer B
- Both answers
Reference answer: Answer A
3 · original EN · What is the main issue with the summary?
Synthetic example · text written for this page.
Synthetic reference: a completed form must contain a project code, a purpose and a requested amount. Candidate summary: the form needs a project code and a purpose. The summary makes no other claims.
Possible answers
- It adds an unsupported requirement
- It omits the requested amount
- It contradicts the project code requirement
Reference answer: It omits the requested amount
Long documents 3 examples
These examples are short, fictional document sets. Documents in this benchmark category contain 3,000–10,000 tokens.
1 · original PL · Can device B be started now?
Synthetic example · text written for this page.
Short, synthetic document set. Section 1 — plan: device A must finish testing before device B starts. Team 1 is responsible for testing; Team 2 for startup. Section 2 — unrelated issue: device C passed testing and was approved for operation. That approval applies only to C. Section 3 — report: measurements for device A are complete, but the approver’s signature is missing. In this process, testing is complete only after approval. Section 4 — sequence: Team 2 is waiting for confirmation that test A is complete. No exception to this rule has been granted.
Possible answers
- Yes, because device C has approval
- Yes, measurements alone are sufficient
- No, test A has not been approved
Reference answer: No, test A has not been approved
2 · original EN · Can the request for Project A be approved?
Synthetic example · text written for this page.
Short synthetic case file.
Section 1 — request: Project A needs 12 units of Material X.
Section 2 — warehouse: 20 units of Material X are stored, of which 6 are reserved for Project B. Reservations cannot be reassigned.
Section 3 — unrelated shipment: 30 units of Material Y arrive tomorrow. They cannot substitute for Material X.
Section 4 — rule: approve a request only if enough unreserved units of the requested material are available now.
Section 1 — request: Project A needs 12 units of Material X.
Section 2 — warehouse: 20 units of Material X are stored, of which 6 are reserved for Project B. Reservations cannot be reassigned.
Section 3 — unrelated shipment: 30 units of Material Y arrive tomorrow. They cannot substitute for Material X.
Section 4 — rule: approve a request only if enough unreserved units of the requested material are available now.
Possible answers
- Yes, 14 unreserved units of Material X are available
- No, only 6 units are available
- Only after the Material Y shipment arrives
Reference answer: Yes, 14 unreserved units of Material X are available
3 · original PL · According to the procedure, what is the status of the current alert?
Synthetic example · text written for this page.
Short, synthetic service report. Part A — procedure: an alert is resolved only after repair and a successful verification test. Part B — archive: the previous alert for module A was resolved after testing. This entry refers to an earlier incident. Part C — current incident: the damaged component in module A has been replaced; the verification test for the new repair has not yet run. Part D — handover note: the next shift must run the missing test.
Possible answers
- Resolved
- Awaiting a verification test
- No action required
Reference answer: Awaiting a verification test
Real documents 3 examples
This section shows fictional documents to illustrate the question types. The actual benchmark uses real documents in this category.
1 · original EN · Which submission method does this document specify?
Synthetic example · text written for this page.
Synthetic document written for this demonstration; it is not an official notice.
Notice from Organisation A: applications for the fictional equipment programme are accepted electronically only. A submission consists of a completed form and an inventory list. A telephone call does not count as a submission.
Notice from Organisation A: applications for the fictional equipment programme are accepted electronically only. A submission consists of a completed form and an inventory list. A telephone call does not count as a submission.
Possible answers
- Telephone only
- Electronic submission with the required attachments
- Any method without attachments
Reference answer: Electronic submission with the required attachments
2 · original PL · What information must be entered in the register when collecting equipment?
Synthetic example · text written for this page.
Synthetic demonstration document; this is not an official document. Notice from Organisation A: the demonstration warehouse is accessible to project teams from 08:00 to 16:00. Collecting equipment requires entering the project code in the register. The person on duty checks the code but does not change the access hours.
Possible answers
- Project code
- Employee’s private address
- Identity document number
Reference answer: Project code
3 · original EN · Which request has been accepted?
Synthetic example · text written for this page.
Synthetic document written for this demonstration; it is not a legal decision.
Decision record of Organisation A: the request to expand the test area is accepted for two workstations. The separate request to expand the storage area remains pending because a floor plan is missing. The record makes no decision about opening hours.
Decision record of Organisation A: the request to expand the test area is accepted for two workstations. The separate request to expand the storage area remains pending because a floor plan is missing. The record makes no decision about opening hours.
Possible answers
- Expansion of the storage area
- Expansion of the test area for two workstations
- A change to opening hours
Reference answer: Expansion of the test area for two workstations
Search and RAG 3 examples
1 · original EN · Which passage directly answers the user question?
Synthetic example · text written for this page.
Synthetic knowledge snippets.
Passage A: a project owner can export reports from the Reports menu.
Passage B: account passwords are changed from Security settings.
Passage C: billing history is visible to the billing role.
User question: where can a project owner export a report?
Passage A: a project owner can export reports from the Reports menu.
Passage B: account passwords are changed from Security settings.
Passage C: billing history is visible to the billing role.
User question: where can a project owner export a report?
Possible answers
- Passage A
- Passage B
- Passage C
Reference answer: Passage A
2 · original PL · Does the retrieved context provide enough information to answer the user’s question?
Synthetic example · text written for this page.
Synthetic knowledge base excerpt: completed tasks can be archived. Archiving preserves the comment history. User question: can an archived task be restored? The retrieved material does not describe restoration.
Possible answers
- Yes, restoration can be confirmed
- Yes, restoration can be ruled out
- No, a passage about restoration is needed
Reference answer: No, a passage about restoration is needed
3 · original EN · How does the claim relate to the source?
Synthetic example · text written for this page.
Synthetic source: notifications can be paused for one hour, four hours or until the next day. Pausing does not delete existing notifications.
Claim to check: pausing notifications deletes all earlier notifications.
Claim to check: pausing notifications deletes all earlier notifications.
Possible answers
- Supported
- Contradicted
- Not addressed
Reference answer: Contradicted
Ticket routing 3 examples
1 · original EN · Which queue should receive the message?
Synthetic example · text written for this page.
Fictional support rules: login failures go to Account Access; questions about charges go to Billing; requests for new capabilities go to Product Feedback. Message from Customer A: I know my password, but the verification code never arrives and I cannot sign in.
Possible answers
- Account Access
- Billing
- Product Feedback
Reference answer: Account Access
2 · original PL · Which queue should receive the main request?
Synthetic example · text written for this page.
Fictional procedure: inability to pay goes to Payments, a missing parcel goes to Delivery, and questions about specifications go to Product Information. Message from Customer A: the specifications are clear and I want to buy the product, but every attempt to pay for my cart ends in an error.
Possible answers
- Payments
- Delivery
- Product information
Reference answer: Payments
3 · original EN · Which priority follows from this policy?
Synthetic example · text written for this page.
Fictional priority policy: service unavailable for all users = urgent; unavailable for one team without a workaround = high; cosmetic issue with a workaround = normal. Report: one team cannot use the export function and no workaround exists. Other teams can use it.
Possible answers
- Urgent
- High
- Normal
Reference answer: High
Rules and deadlines 3 examples
1 · original PL · What delivery charge follows from these rules?
Synthetic example · text written for this page.
Fictional price list: standard delivery costs 10 points. It is free when the value of the products after discounts is at least 100 points. Customer A chose standard delivery. The products cost 120 points before a 25% discount.
Possible answers
- 0 points
- 10 points
- 25 points
Reference answer: 10 points
2 · original EN · Is the pass valid on the immediately following Friday?
Synthetic example · text written for this page.
Fictional access rule: a temporary pass is valid for the three calendar days after the day it is issued. The issue day is excluded. A pass was issued on a Monday. There are no exceptions.
Possible answers
- Yes
- No
- The issue hour is required to decide
Reference answer: No
3 · original PL · Does task A meet the conditions for automatic assignment?
Synthetic example · text written for this page.
Demonstration rules: a task is assigned automatically only if it has an owner and at least two independent approvals. Task A has an owner, one approval and one comment without approval.
Possible answers
- Yes
- No, the second approval is missing
- No, the owner is missing
Reference answer: No, the second approval is missing
Scale-based assessment 3 examples
1 · original PL · Which level on the scale matches the description?
Synthetic example · text written for this page.
Fictional impact scale: 0 — no disruption; 1 — minor disruption with a workaround; 2 — a task blocked for one team; 3 — a core service blocked for everyone. Team A cannot complete its task and has no workaround. Other teams are working normally.
Possible answers
- 0 — no disruption
- 1 — minor disruption
- 2 — one team blocked
- 3 — everyone blocked
Reference answer: 2 — one team blocked
2 · original EN · Which completeness level applies?
Synthetic example · text written for this page.
Synthetic completeness rubric: 0 = none of the required fields; 1 = one field; 2 = two fields; 3 = all three fields. Required fields are purpose, project code and requested amount. The form contains a purpose and a project code, but no amount.
Possible answers
- 0
- 1
- 2
- 3
Reference answer: 2
3 · original EN · What relevance level fits the passage?
Synthetic example · text written for this page.
Fictional relevance scale: low = unrelated topic; medium = same topic without answering the question; high = directly answers the question. Question: how can notifications be paused? Candidate passage: open Notification Settings and choose Pause, then select a duration.
Possible answers
- Low
- Medium
- High
Reference answer: High
Previous release results and methodology: basal-1.0 technical report ↗
See models in use