basal engine 2.5: many GPUs, a model cascade and the same answers

A safe upgrade

The rule of this release is simple: if you do not change your configuration, you get the same probabilities as in 2.4.1. The release notes list four changes on the default path. They concern how requests are handled, not the decisions themselves:

• Question names with ::. In a request with task questions, a question named <task>::<part> is refused (HTTP 422). Before, the task could read that question’s answer as its own (#59).
• Clearer errors. An error body has the field code (INVALID_REQUEST, OUT_OF_MEMORY, INTERNAL_ERROR), and a 422 response also has detail. The field error keeps its text (#62).
• More in GET /v1/models. The model list gives the certified thresholds for each max_error target and question type, and the context settings (#64).
• A request ID. Every response carries the header x-request-id (#68).

One default change speeds things up without changing results. The server tokenizes the part the prompts share, the template and the state, only once. The engine gets the same token ids as before. A rank request with 300 candidates over a 12.5k-token state takes 0.14 s of CPU instead of 5.8 s.

There are fixes too. An empty "questions" now gives HTTP 422 instead of 500. Edited chat states with the state cache no longer end in an HTTP 500 with NaN values. On basal-omni-1.0-ultra the bug hit 27 of 1,248 such requests; after the fix, none. Version 2.4.1 had the same problem.

Many GPUs and a model cascade

basal-router puts several basal-serve replicas behind one port. The --gpus 0,1 option starts one replica per GPU and stops them with the router, also when the router is killed. Requests over the same state go to the same replica (state-hash routing with bounded load), so its state cache keeps working. A replica that answers 503 or not at all gets no requests until it is ready again.

On two H100s the router served 1.58× the requests of one GPU, with twice the clients. Scaling is not linear yet: two GPUs do not do twice the work for now.

A cascade (--escalate, --escalate-below) asks a bigger model again when the top probability is under a threshold. Levels can be chained, for example basal-omni-pico → mini → ultra, each with its own threshold:

basal-serve --model Remek/basal-omni-pico --port 8001 &
basal-serve --model Remek/basal-omni-1.0-mini --port 8002 &
basal-serve --model Remek/basal-omni-1.0-ultra --port 8003 &
basal-router --replicas http://127.0.0.1:8001 \
    --escalate http://127.0.0.1:8002 --escalate-below 0.8 \
    --escalate http://127.0.0.1:8003 --escalate-below 0.9 --port 8000

Every answer says which model gave it (answered_by), and with two or more levels it also gives the level. /metrics counts the answers at each level. When a level is down, the previous level’s answer stays and the request does not fail. The --route option sends states in one script, such as Cyrillic, to their own replicas.

On a held-out half of the set, the mini → ultra cascade at a threshold of 0.95 served about 1.5× the requests of ultra alone on one H100, at the same accuracy (within 0.5 pt). Ultra gets only the questions mini is unsure of.

Blend (--blend) works the other way round. Every question goes to several models, and the router returns the weighted mean of their probabilities. A blend of 0.6 ultra + 0.2 mini + 0.2 pico scored +0.45 pt over ultra alone (95% interval: +0.09 to +0.81). It is an option for quality, not for speed.

The LLM fallback (--fallback) comes after the last level. A choice question whose top probability is still under a threshold T goes to a chat model you serve behind an OpenAI-compatible endpoint. Its answer ("answered_by": "fallback") replaces the basal answer. In an example with basal-omni-1.0-ultra and Gemma 4 31B-it with thinking at T = 0.8, the score rose by 1.40 pt (95% interval: +0.90 to +1.89), and 8.4% of the questions went to the chat model. This result assumes no call was cut by the timeout. One call took 54 s at the median and 308 s at p95. With the default --fallback-timeout 60 the basal answer stayed in 46 of 100 calls, which gave about three quarters of the gain on that sample.

The cascade, blend and fallback measurements come from a held-out half of the set. Details: RESULTS.md and SERVING.md.

Answers you can rely on

basal returns calibrated probabilities. Version 2.5 lets an application use them directly, without its own threshold logic:

• Answer sets. "coverage": 0.9 returns the set of options that holds the right one with that probability. When the model hesitates between two categories, the set holds both, and a person chooses only between them.
• Abstaining. If you set "max_error", a choice, yes/no or score answer gets "abstain": true when the confidence is below the certified threshold for its type. The answer itself does not change, but the application knows this case is better passed to a person.
• A second read (--reread T). Answers below the threshold T are read again, with the state given twice, and the two reads are combined. --reread-image reads an image again at more tokens.
• Option order. basal-bench --order-check measures how often the top answer changes when the options are reversed or rotated. --order-average K returns the mean over K orders and reads the state only once. It helps basal-omni-1.0-mini only: +0.6 pt top-1 on our development set with K = 2, at about twice the time per question. On max and pico it does not help, so it is off by default.

The ready question sets guard_input, guard_output and route_intent have hand-picked thresholds for now, not certified ones. With max_error they use the served model’s calibration for each question type.

More hardware and a faster start

basal-omni-1.0-ultra, the largest model of the family, gets three modes that quantise it at load time, and it fits on smaller cards. They work for ultra only. The other basal-omni models failed the accuracy check, so the engine refuses them in these modes.

Checkpoints in -FP8 repositories now load without a bf16 copy on the GPU. As a result basal-omni-1.0-ultra-FP8 runs on one 80 GB H100; before, it ran out of memory at load. --state-cache-bits 8 keeps cached states as 8-bit integers: about 1.7× as many fit per GB, and on basal-omni-1.0-max 99.45% of top answers are the same as with the 16-bit cache.

A faster start. basal-prepare saves the kernels tuned for your GPU and engine version once, and --prepared auto loads them at start-up. The answers are byte-identical to a cold start. basal-omni-pico starts in 72 s instead of 287 s. The model repositories already carry the directory prepared/ for the H100, and --prepared auto fetches only the part for your GPU and versions. First start on an H100 with empty caches: basal-omni-1.0-mini 290 → 172 s, basal-omni-1.0-max 259 → 164 s. Pico on the H100 now also gives the same answers after every start; in 2.4.1 its fast-mode answers could differ between starts.

Apple Silicon. The state cache now works in --mode mlx, with answers bit-identical to running without the cache. On a MacBook Air M4, a cached 2.1k-token policy cut the time per request from 13.7 to 0.9 s. The opt-in --mlx-forward basal runs the last layers at the answer rows only: basal-omni-1.0-mini-MLX answers one prompt in 131 ms instead of 225 ms (M4 Pro). basal-omni-pico runs on a Mac in --mode mlx. On 1,700 held-out prompts its answers match CUDA eager float32 (top-1 100%).

Older GPUs. The guide OLD_GPUS.md runs basal-omni-pico on GTX 10xx, RX 480 / 580 and Intel Iris Xe through llama.cpp’s Vulkan backend. basal-selfcheck compares your setup with the reference answers shipped with the release.

In production: a live page, metrics, API keys

A live web page for basal-serve (--ui on, off by default) shows the server’s status, traffic since start, the state cache, GPU and host telemetry, the configuration and the metadata of the last 200 requests, never their text. --ui full also shows each of those requests and its answer. With one page open the cost is small. On an H100 with basal-omni-1.0-mini the throughput stayed within 1% of --ui off. The median latency at 8 clients rose by 0.4–2.2% with 1–2 pages and by 5.2% with --ui full. At one client the answers are byte-identical.

One page for the router and all its replicas (basal-router --ui on) shows routing, readiness, requests in flight, failures and the router’s last 200 requests. Next to them are cards for the replicas with their status, traffic, state cache and GPU telemetry. On 2 × H100 the median at 8 clients rose by 1.3%, within the run-to-run spread of --ui off.

• GET /metrics in Prometheus format for basal-serve and basal-router, with the engine version, state-cache evictions and prefix hits (#48). While the model loads, the server answers HTTP 503, and a failed start-up ends the process instead of hanging.
• API keys (--api-keys PATH, opt-in). A JSON file holds SHA-256 digests of the keys. Every route except GET /health needs a key, and an admin sees key use at GET /v1/keys. --cors-origins opens browser access for the listed origins (#68). basal-router passes a client’s key on to the replicas, but never to --fallback.
• Replicas stay local. Replicas the router starts with --gpus listen on 127.0.0.1. Before, they listened on 0.0.0.0, so the network could reach them past the router.

Integrations: from the OpenAI API to LM Studio

• OpenAI Decisions API. basal-serve answers /v1/decisions, so client.decisions.create from the OpenAI SDK works with a local model. The probabilities are the same as on /v1/systemone.
• LM Studio (opt-in). basal-export-gguf --systemone turns a basal GGUF file into an LM Studio decision model: choice, score and yes/no, up to 26 options, with calibrated temperatures and no basal-serve. It needs LM Studio 0.4.26 or newer (beta channel) with llama.cpp engine 2.65.1 or newer. On all 607 questions LM Studio answered, the top answer was the same as in basal-serve. LM Studio stops a request after 30 s.
• TypeScript client (clients/typescript, no runtime dependencies): typed questions and answers, answer sets, abstain, ask_if / depends_on, images and errors. For now you build it from the repository; it is not on npm yet.
• Agent frameworks. Thin tools for LangChain, LlamaIndex and CrewAI use the HTTP API (basal.adapters). Ready question sets (triage, moderation, compliance) are in basal-run --preset.
• MCP. A server with several models offers list_models and the argument model for the tools decide and decide_many (#66).
• The Python client has a method for every question type, AsyncBasal for asyncio and errors by status code (#58).
• Your own models. basal-attach serves a chat model that was never trained on basal, such as Gemma 4 or Qwen, as a decision model. It starts uncalibrated; basal-calibrate fits it on your labelled requests. basal-finetune trains a LoRA on your own labelled decisions. We tested it on Apple Silicon with basal-omni-pico, not yet on other GPUs.
• LoRA adapters at load (--adapter DIR on basal-serve, basal-calibrate and basal-bench). The answers equal, bit for bit, those of the merged checkpoint written by basal-fuse-adapter (#52).

Experimental: rank and basal-navigate

rank scores thousands of candidates in one request. It first screens them with yes/no questions, then picks one of the top 20. One request holds up to 10,000 candidates. basal-navigate retrieves by walking a document tree level by level: topics, pages, passages. It needs no index.

With basal-omni-1.0-max on one H100, against a dense 8B embedding model: MultiDoc2Dial R@1 44.3 vs 34.7 (+9.6), ConditionalQA Hit@1 63.5 vs 44.3 (+19.2).

This accuracy has a price: time. With one query at a time, a search takes p50 3.0 s and p95 3.5 s per MultiDoc2Dial turn, and p50 9.2 s and p95 18.4 s per ConditionalQA question. At 8 concurrent clients the median is 24 s and 64 s. That is why both tools are experimental. --split-at N searches trees larger than the 10,000-candidate limit, but it is not faster: on Natural Questions (2.68 million paragraphs) it takes 51.6 s per question. Details: NAVIGATE.md.

Honest about the limits

• The cascade and the slowest answers. The mini → ultra cascade serves more requests, but at 16 clients its p95 is higher than ultra’s alone.
• basal-navigate is slow: seconds per query with one client and tens of seconds with eight.
• Strix Halo and Apple M5 are not verified on our hardware. --mode strix is maintained by the community. On a Radeon 8060S one short question now takes 161 ms instead of 390 ms, but we have not checked this on our own hardware. We have not yet checked pico’s float32 setting for the Apple M5 on that chip; on an M4 Pro the answers are identical to before.
• Ultra in block FP8 runs without the state cache until 2.5.1. --state-cache-gb is then ignored with a warning at start-up, and requests are served as without the cache.
• Blend and LLM fallback answers are not calibrated. Refit the thresholds with basal-calibrate before you automate them. Fallback answers repeat only with the endpoint settings described in SERVING.md.
• A rare HTTP 500 with the state cache. With --mode fast and --state-cache-gb a rare race can fail one request with an HTTP 500: about 1 in 8,000 in our tests. The server keeps serving, and the request can be retried. The problem exists since 2.3.1. --state-cache-lock on removes it at a cost: on an H100 with basal-omni-1.0-max, 8 clients and 8,740 requests, 0 failures instead of 1, but 8.19 instead of 8.88 requests/s (-8%) and a median of 630 instead of 577 ms.
• Long states on ultra. The same 24k–32k-token request, sent again, gave bit-identical probabilities in 95 of 100 cases, as in 2.4.1. Mini: in 100 of 100.

Thank you, community

A large part of this release came in pull requests. Thank you:

• Miłosz Kucharski (@Butterski): less CPU work in the CUDA cascade (#44), a torch.compile cache that survives a reboot (#47; the first start of basal-2.0-mini after a reboot 417 → 238 s on an RTX 4060 Ti) and --compile-scope layer (#49).
• Paweł Kiszczak (@pawelkiszczak): development commands and pre-commit checks (#46), LoRA adapters (#52), error codes (#62), thresholds and context in /v1/models (#64), model selection in MCP (#66), API keys (#68), calibration on Hugging Face snapshots (#71) and basal-viz with facts and for Gemma 4 (#73, #75).
• Marcin Brzeżański (@marcinbrzezanski): new metrics (#48) and a state cache that keeps the branches of a history (#56). On agent histories it reuses 1.6–1.7× as many prefix tokens.
• Dariusz Koryto (@dkoryto): CI on Linux and macOS (#53), the Python client (#58), the fix for names with :: (#59) and basal-interpolate (#60).
• Grzegorz Stark (@gstark0): the state cache on Apple Silicon (#50).
• Michał Kardyś (@kardysm): --mode strix without waiting for kernel compilation on the first requests (#51).
• Grzegorz Brzezinka (@agentGreg): tests on Apple M5 (#76) and basal-viz on the MLX engine (#78).

We also thank Miłosz for the earlier #17: thanks to it, the language of the state is detected once per request, not for every question.

What 2.5.1 brings

• Certified thresholds for the ready question sets; today they are hand-picked.
• int8 on Apple M5.
• The state cache in block FP8 (--mode fp8).
• A fix for the rare HTTP 500 with --mode fast and the state cache, without the throughput cost of --state-cache-lock.

Getting started

NVIDIA GPU, with uv in a fresh environment:

uv venv --python 3.12 --allow-existing ~/basal-env && source ~/basal-env/bin/activate
uv pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu128
uv pip install "basal[fp8] @ https://github.com/rkinas/basal/archive/refs/tags/v2.5.0.tar.gz"
basal-serve --model Remek/basal-2.0-4.5B --mode fast --template-prefix off --port 8000

Apple Silicon:

uv venv --python 3.12 --allow-existing ~/basal-mlx && source ~/basal-mlx/bin/activate
uv pip install "basal[mlx] @ https://github.com/rkinas/basal/archive/refs/tags/v2.5.0.tar.gz"
basal-serve --model Remek/basal-2.0-4.5B-MLX-8bit --mode mlx --port 8000

Two GPUs behind one port, with the live page for the router and the replicas:

basal-router --gpus 0,1 --port 8000 --ui on -- --model Remek/basal-omni-1.0-mini --mode fast --ui on

To get an answer set and the abstain signal, give the question the fields coverage and max_error:

{
  "state": "Customer report: the wardrobe I ordered arrived with cracked doors. Please replace the doors by Friday.",
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle the case?",
      "criteria": {
        "service": "Repairs and replacement parts",
        "sales": "New orders",
        "accounting": "Billing"
      },
      "coverage": 0.9,
      "max_error": 0.01
    }
  }
}

This is an example request; we do not simulate its results. The coverage levels come from a calibration fitted with basal-calibrate --conformal. When the calibration file does not have a level, the set holds every option and the field coverage_note says why.

After the upgrade the default answers do not change. Still, measure each new option on your own requests before it goes to production.

Full release notes and measurements → · Inference server, REST and MCP →