New release19 October 20267 min read

basal 2.2 engine: calibration on your own data

basal-calibrate in engine 2.2 checks on your own labelled examples whether the model’s confidence means what it says, and fits certified thresholds. The model decides alone only where the error target holds and hands the rest to a person. Plus FP8 and GGUF ports of basal-omni-1.0-mini and max.

Remigiusz Kinas · ai5

A decision model is only useful when “90% sure” means 90% on your data. The calibration shipped with the models was fitted on our data. On yours the model can be confident in a different way, even when it is just as accurate.

The basal 2.2 engine lets you check and fix that yourself. basal-calibrate measures the model on your labelled examples and fits certified thresholds. The model decides alone only where the error target holds. It hands the rest to a person. We are also releasing FP8 and GGUF ports of basal-omni-1.0-mini and max.

8.0%wrong decisions at the shipped file’s “1%” setting, on a new domain

0.268 → 0.083calibration error (ECE) of choice questions after calibrating on the user’s data

4 portsFP8 and GGUF of basal-omni-1.0-mini and max, text only

What “90% sure” means

Every basal model returns the probability of each allowed answer. In act and tool questions you set an error target, for example "max_error": 0.01. The model acts alone only when its confidence reaches the threshold certified for that target. Otherwise it hands the decision to a person.

The temperatures and thresholds are in the CALIBRATION.json file shipped with the model. We fitted them on our data. On your domain the model may be confident in a different way. A threshold from our data then accepts too few decisions, or too many, wrong ones included.

You cannot see this without measuring. So the first step in 2.2 is a report: how the file you use today does on your labelled data. A hosted decision API offers no such step: its confidence is what it is on your data. A basal model runs on your side, so you can calibrate it on your own data.

Calibration on your own data

You need a JSONL file: one /v1/systemone request per line, with the correct answer to each question in "gold". Send the requests exactly as in production: with the same options, facts and language of the state.

JSONL · ONE LABELLED EXAMPLE

{"id": "t-17", "group": "customer-17", "state": "...",
 "questions": {
   "dept":  {"type": "choice", "instructions": "...", "criteria": {"returns": "...", "it": "..."}, "gold": "returns"},
   "late":  {"type": "noul",   "instructions": "...", "gold": "yes"},
   "level": {"type": "score",  "instructions": "...", "criteria": ["low", "mid", "high"], "gold": 2}}}

TERMINAL · MEASURE, FIT, SERVE

basal-calibrate labelled.jsonl --model Remek/basal-omni-1.0-max --mode eager --report-only
basal-calibrate labelled.jsonl --model Remek/basal-omni-1.0-max --mode eager
basal-serve --model Remek/basal-omni-1.0-max --mode eager --calibration CALIBRATION.custom.json

The first command only measures and writes nothing. The second writes CALIBRATION.custom.json, fitted by the same method as the files shipped with the models. The third starts the server with it.

Split. The decisions are split into a part for fitting (70%) and a held-out part (30%). Decisions of one group, such as one customer, stay on the same side.
Temperatures. One per question type (choice, noul, score). They correct how the model’s confidence matches its accuracy on your data.
Certified thresholds. The threshold for a 1% or 5% target is the lowest confidence at which the upper bound of the error of accepted decisions (one-sided 95% Clopper–Pearson) stays within the target. If no confidence meets it, there is no threshold and basal-serve hands every such decision to a person.
Check. The held-out part, not seen in fitting, shows how many decisions the file automates and at what error, next to the file you use today.

Calibrate with the engine, port and quantisation you run in production. The probabilities, and so the thresholds, belong to them. Calibration works with every basal-2.0 and basal-omni model.

The whole process, step by step from labelled examples to the server, is shown in this film (7:16, Polish voice-over, PL and EN subtitles). It is also in Basal Lab: Confidence checked on your data.

FILM · POLISH NARRATION + MUSIC · PL/EN SUBTITLESbasal-omni-1.0-mini · engine 2.2

Six steps: labelled examples, the split into fitting and control, a report without writing anything, fitting the calibration file, a check on new examples, and the server with the new file. Download the film ↓

Measured on a new domain

We checked this with basal-omni-1.0-mini (bf16, engine 2.2, 16 October) on the open typed-decisions dataset (Apache-2.0). These are synthetic decisions over JSON records in four workflows: customer service, invoices, security incidents and agent traces. The labels come from a teacher model. The model was not trained or calibrated on this data.

Calibrated on the train split: 3,958 labelled decisions in 800 requests, plus 596 classify questions and 1,600 verify claims. Measured on the held-out part of the test split: 172 choice, 184 yes/no and 237 score decisions.

basal-omni-1.0-mini on typed-decisions · test split · file fitted on the train split
On the test split	File shipped with the model	File calibrated on the train split
ECE: choice / yes-no / score	0.268 / 0.106 / 0.104	0.083 / 0.073 / 0.052
`max_error: 0.01`: automated, and the error	12.6%, of which 8.0% wrong	0% (nothing meets the target)
`max_error: 0.05`: automated, and the error	54.3%, of which 21.4% wrong	3.7%, no errors
`verify`: claims at `max_error: 0.01`	no task threshold	54.8%, no errors (upper bound 1.1%)

The shipped file misses its target on this domain. The “1%” setting would automate one decision in eight, and 8.0% of them would be wrong. The “5%” setting would automate over half of the decisions at 21.4% error. On our calibration data the same file meets its target.
After calibration the ECE falls for every type: more than threefold for choice, by half for score and by about 30% for yes/no. The temperatures go up (choice from 1.14 to 2.67): on these records the model is overconfident.
Almost nothing is automated, and that is the right answer. mini’s accuracy on this data is 0.57 / 0.80 / 0.67 (choice / yes-no / score), against noisy labels. At that accuracy no threshold meets the 1% target, so the decisions go to a person.
Where the model is accurate, automation stays. On verify claims mini is 95% accurate. The threshold fitted on the train split automates 54.8% of the test claims, without a single error.

All numbers are on the held-out part of the test split, not seen in calibration. Description and commands: CALIBRATION.md.

What else is in engine 2.2

Task thresholds from your labels. Labelled classify, verify and extract questions certify the automated flag in the answers of these tasks, by the same method as the decision thresholds. Without such a threshold the flag is null.
extract reports automated for every field, like a classify answer and each claim of verify.
A few dozen labels to start. When a type has fewer than 50 decisions for fitting but at least 10, its temperature is blended with the shipped one in proportion to the number of labels. 50 decisions for fitting, about 72 labelled ones, fit it fully.
The report as served. You see how many decisions today’s file automates on your data and at what error, each type at the threshold the server actually applies.
A file knows its model. The fitted_on field records the model and its Hugging Face revision. basal-serve warns when a file comes from another model or revision, and with --strict-calibration it refuses to start.
Fix. A looser target (5%) no longer keeps a stricter threshold than the 1% target. Before, the scan could stop early and automate fewer decisions at 5% than at 1%.

The rest is unchanged. Every other request takes the same code path as in v2.1.0. The model weights do not change. Full list: WHATS_NEW.md.

FP8 and GGUF for basal-omni-1.0-mini and max

Four new repositories with the same model and the same API, in smaller formats. All of them read text only; for images and audio use the bf16 models.

FP8 through vLLM. Less memory (mini 12 GB instead of 16 GB, max 14 GB instead of 24 GB) and 1.4–1.5 times more decisions per second than bf16 through vLLM: mini 23.2 against 15.4, max 8.7 against 5.6 (one RTX PRO 6000, 16 requests in flight, one question over a state of about 1,500 tokens). Needs an Ada, Hopper or Blackwell GPU.
GGUF Q8_0 and Q6_K through llama.cpp or Ollama. They run on NVIDIA GPUs, on Apple Silicon Macs and on the CPU. mini Q8_0 runs on a 16 GB Mac; max Q8_0 with a 16,384-token context does not fit a 24 GB Mac (it runs with 8,192); use a 32 GB Mac or the Q6_K file.

Agreement with the bf16 model · 10,606 decisions of the development set · one RTX PRO 6000
Port	Size	Same answer as bf16 (2–10 options)	Accuracy against bf16	Automated at 1% / 5%
mini-FP8 · vLLM	12 GB	96.2%	−0.35 pt	41.9% / 63.5% (bf16: 41.7% / 77.6%)
max-FP8 · vLLM	14 GB	96.9%	−0.34 pt	45.4% / 82.5% (bf16: 47.2% / 83.0%)
mini Q8_0 · GGUF	7.9 GB	98.7%	−0.06 pt	42.4% / 78.3%
mini Q6_K · GGUF	6.2 GB	97.1%	+0.05 pt	Q8_0 calibration file
max Q8_0 · GGUF	12.7 GB	98.7%	−0.07 pt	47.9% / 82.5%
max Q6_K · GGUF	9.8 GB	97.6%	−0.11 pt	Q8_0 calibration file

Every port passes the basal port gate: agreement with bf16, agreement where bf16 is sure, accuracy within 1 point, and ECE. The changed decisions are mostly near-ties of two options. Q5_K_M and Q4_K_M did not pass the gate, so we do not ship them.
Every port has its own calibration (CALIBRATION.vllm.json or CALIBRATION.gguf.json), fitted on its own readouts. Automation is measured on the calibration test set (8,560 decisions). With GGUF the observed error was 0.7% / 3.9% for mini and 0.5% / 4.0% for max.
Mac and Ollama. mini Q8_0 on an Apple M4 Pro: the same answer as bf16 in 98.9% of decisions, 1.2 decisions per second. Through Ollama 0.35: 98.0%.
MLX on Apple Silicon Macs. basal-omni-1.0-mini-MLX: 8.6 GB, the same answer as bf16 in 98.6% of decisions (2–10 options), 0.73 decisions per second (Apple M4 Pro, 24 GB), automated 34% / 74% of decisions at the 1% / 5% target; basal-omni-1.0-max-MLX: 13.6 GB, the same answer as bf16 in 98.9% of decisions (2–10 options), 0.25 decisions per second (Apple M4 Pro, 24 GB; with 8 long prompts in flight it can run out of memory), automated 36% / 85% of decisions at the 1% / 5% target; basal-omni-1.1-max-MLX: 13.6 GB, the same answer as bf16 in 98.4% of decisions (2–10 options), 0.27 decisions per second (Apple M4 Pro, 24 GB; with 8 long prompts in flight it can run out of memory), automated 35% / 80% of decisions at the 1% / 5% target. Text only.
Availability. basal-omni-1.0-mini-FP8, basal-omni-1.0-max-FP8, basal-omni-1.0-mini-GGUF, basal-omni-1.0-max-GGUF.

The models’ Werdykt scores (mini 0.769, max 0.823) are for the bf16 weights; the ports are not scored on it separately. Details: OMNI.md.

Limitations

Calibration does not improve accuracy. It makes the model’s confidence tell the truth. A model that is weak on your data hands more decisions to a person after calibration.
The 1% target needs many labels. A few hundred labelled decisions usually certify the 5% target. The 1% target needs at least 299 accepted decisions without an error, in practice about 1,000 examples or more. With fewer, use the new temperatures and the 5% threshold, or route uncertain cases to a person.
The thresholds hold for what you calibrate on. They hold for that mix of question types and prompts, and for your labels. When the process or the data change, measure again with --report-only.
The ports read text only. With vLLM an answer close to a tie can change between runs; VLLM_BATCH_INVARIANT=1 gives repeatable answers at about 23% fewer decisions per second. At the 5% target mini-FP8 automates less than bf16. Ollama reads at most 20 options and 4,096 tokens.
Keep oversight of consequential decisions. Decisions with serious consequences for people require human oversight.

Community contributions

basal is open. People build their own tools on it, and we bring their work into the engine. Earlier, in basal 1.5, Paweł Kiszczak (@pawelkiszczak) brought support for Apple Silicon (MLX, MPS), llama.cpp and Ollama (PR #4); many thanks for that. For 2.2, thank you to:

Piotr Durlej (@pdurlej), basal-export-gguf: converts or repairs GGUF files so that their tokenizer matches the official -GGUF files (PR #8).
Kuba Sołtys (@qooba), --mode tpu: an experimental JAX / XLA backend that runs basal-2.0 on Google TPU (PR #15).
Michał Kardyś (@kardysm), --mode strix: an experimental mode with Triton kernels for AMD Strix Halo, for basal-1.5 and basal-2.0 (PR #14).
@lukaspoloki, basal-viz: a local page that shows every step of a decision of the basal-2.0 models (Llama architecture); the command of the same name starts it (PR #2).

Both new backends are experimental and maintained by the community. The numbers in their documentation are their authors’ own measurements. Code, issues and pull requests: github.com/rkinas/basal.

Getting started

TERMINAL · NVIDIA GPU · ENGINE 2.2 WITH BASAL-OMNI

uv venv --python 3.12 --allow-existing ~/basal-env && source ~/basal-env/bin/activate
uv pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu128
uv pip install "basal[media] @ https://github.com/rkinas/basal/archive/refs/tags/v2.2.0.tar.gz"
basal-calibrate labelled.jsonl --model Remek/basal-omni-1.0-mini --mode eager --report-only

TERMINAL · FP8 THROUGH VLLM · TEXT ONLY

uv pip install vllm==0.31.0 "basal @ https://github.com/rkinas/basal/archive/refs/tags/v2.2.0.tar.gz"
basal-serve --model Remek/basal-omni-1.0-max-FP8 --mode vllm --max-len 16384 --gpu-memory 0.9 --port 8000

TERMINAL · GGUF THROUGH LLAMA.CPP · ALSO ON A MAC

uv pip install "basal[gguf] @ https://github.com/rkinas/basal/archive/refs/tags/v2.2.0.tar.gz"
hf download Remek/basal-omni-1.0-mini-GGUF --local-dir basal-omni-1.0-mini-GGUF
basal-serve --model ./basal-omni-1.0-mini-GGUF --mode gguf --gguf basal-omni-1.0-mini-GGUF/basal-omni-1.0-mini-Q8_0.gguf --port 8000

Engine v2.2.0 is on GitHub: release notes. On NVIDIA, build llama-cpp-python with CUDA (see the engine’s GGUF.md); on a Mac it uses Metal. Server options, including --calibration: the inference server page.