basal engine 2.3: the same model, up to 17× the throughput

The short version: where the speed comes from

From Franklin’s “time is money” to a faster engine: 43 seconds on shared context and two measurements. Narration is in Polish; switch subtitles to Polish or English.

Quotation: Benjamin Franklin, “Advice to a Young Tradesman”, 1748. Portrait: Henry Wolf, 1898, Smithsonian American Art Museum, public domain.

One context, many questions

In 2.2, the fast CUDA-graph engine supported Bielik-based basal-2.0 models. The Gemma 4-based basal-omni family used eager or a separate vLLM integration. In the previous eager path, each question processed the entire state again.

In 2.3, the fast engine also supports Gemma 4. It computes the state representations — the keys and values used by attention — once per request, then processes all questions and option orders over them. Sliding-window layers preserve the context window each question would see in a separate request.

1. The application sends one state and multiple questions. All questions refer to the same document.
2. The engine computes the shared part. It avoids repeating preparation of the same context for every question.
3. The model returns separate decisions. Each question retains its own type, options and result.

This benefit applies to questions in one request. Sending five separate requests is not the same as putting five questions alongside one shared state.

H100 results: single and multiple questions

Both measurements use the same GPU and performance harness. A single question uses a state of about 1,500 tokens. In the second profile, each request contains five questions over a 3,000–10,000-token state.

16 requests in flight. Exception: in 2.2, ultra fitted 4 requests for a single question and 2 for five questions; higher concurrency ran out of memory. The ultra gain therefore also includes the ability to sustain more concurrency in 2.3. Ratios come from the underlying measurements; table values are rounded.

Throughput is not individual response latency. 61.2 decisions/s measures the server’s work rate at the stated load. It does not mean each request has a latency of 1/61.2 seconds. Likewise, 4.8 questions/s with five questions per document means about 0.96 such requests/s.

We separately measured the prebuilt mini-FP8 checkpoint: 80.6 decisions/s for a single question. That is 8.0× the BF16 eager mini baseline. Because the weight format also changes, we keep it outside the main BF16 comparison.

On RTX PRO 6000 Blackwell Max-Q, mini reached 5.6 → 23.2 decisions/s and max 1.0 reached 2.3 → 5.0. Five-question throughput was 0.85 → 18.4 and 0.37 → 5.3 questions/s respectively. In 2.3, this path is explicitly enabled with --mode fast --gemma-attn flex, pending completion of its calibration check on that GPU. It is not yet the default Blackwell configuration.

What changed in the computation

• Only the answer positions needed. In mini, the last 18 of 42 layers reuse previously computed keys and values. We compute them only at the positions where answers are read — about 40% less computation per decision. In max and ultra, a similar optimisation applies to the last layer.
• Attention matched to the architecture. On Hopper, FlashAttention 3 handles sliding-window layers, while FlexAttention handles global layers with a head width of 512. These replace the slow reference path.
• Less execution overhead. We fuse q/k/v and gate/up projections, compile layers and replay computation in CUDA graphs. Compact batches reduce unnecessary padding.
• Less context memory. Sliding-window layers retain only the final window of keys and values. For ultra at 10,000 tokens, that part of the cache falls from 9 to 1.6 GB. This is not total VRAM usage. The cuBLAS workspace per graph falls from 32 to 8 MB.

What about images, audio and vLLM?

The fast mode above processes text. Images, audio, evidence and deterministic operation use the eager path. It now also uses the engine’s own Gemma 4 forward pass: state and media encoders run once per request, instead of once per question.

In the max measurement with three questions about one image, median latency fell from 204 to 131 ms, about 36% lower. This is a specific multimodal latency measurement. The 17× figure from ultra’s text test does not apply to it.

Within eager, mini rises from 10.0 to 17.2 decisions/s for one question and from 1.56 to 15.3 questions/s for five questions. Max rises from 0.72 to 4.5 questions/s in the latter profile. Optional --eager-attn auto accelerates attention on Hopper. For mini, --ple-offload on moves the per-layer embedding table to host RAM: it saves 5.2 GB of VRAM at about a 4% throughput cost.

• vLLM: faster mini prefill. Skipping unneeded positions in shared layers is enabled by default. On H100: 41.5 → 60.8 decisions/s for one question and 24.2 → 39.0 questions/s for five. Here the baseline is the previous vLLM path, not eager.
• vLLM: media in mini. Images and audio are supported through --mode vllm --media on. Tests at 8 concurrent requests achieved 6.0× and 7.9× the eager throughput respectively. Top-answer agreement was 96.5% for images and 98.3% for audio; 100% on the high-confidence subset. Media in 1.1-max through vLLM remains blocked because it did not meet the required agreement check.
• vLLM: repeatable probabilities. --vllm-preset deterministic produced bit-identical results across two 1.1-max runs on 10,606 decisions. Throughput was 0.64× the normal profile for one question and 0.22× for five. Reproducibility is a deliberate trade-off, not a free speed gain.

Speed with quality checks

When introducing faster BF16 computation, we checked more than decisions per second. Across 10,606 development decisions with one option order, accuracy changed by at most 0.06 percentage points. Top-answer agreement was about 98.9–99.4% overall and 99.85–99.99% on the subset with a margin of at least 0.2. These values measure agreement between execution paths, not answer accuracy.

When options are nearly tied, different BF16 rounding can change the winner. We do not promise bit-identical results in fast mode. Fast uses the model’s shipped CALIBRATION.json. In the check, automation coverage at a 1% error target changed by less than 0.1 percentage points, and observed error remained below the 1% and 5% targets. This is a result on the checked dataset, not a guarantee for every new domain.

After upgrading, recheck your own requests and calibration. The tools introduced in engine 2.2 remain available.

Fixes and community contributions

Omni no longer fails at GPU startup when --mode is omitted. Hopper with FlashAttention 3 selects fast; other GPUs select eager. We also extended protection against a known PyTorch attention bug to all models on older library versions. Served basal configurations were already protected in 2.2.

Milosz’s contribution (@Butterski, #17) removes another repetition: prompt language is detected once per request. For a compress test with 40,000 characters and 326 questions, this step fell from 509 to 2 ms. That is preparation time, not total inference time.

The speedups in this article concern the basal-omni family. Bielik-based basal-2.0 text models retain their existing fast path. A separate 2.3-versus-2.2 measurement for Bielik is planned for 2.3.1. FP8 KV-cache experiments and load-time weight quantisation that failed the requirements were also left out of 2.3.

How to run it and benefit from the change

Install release 2.3.0 in an environment meeting the documented GPU requirements. Example for 1.1-max:

uv pip install "basal[media] @ https://github.com/rkinas/basal/archive/refs/tags/v2.3.0.tar.gz"

basal-serve --model Remek/basal-omni-1.1-max --port 8000

basal-serve --model Remek/basal-omni-1.1-max --mode eager --media on

The first server command selects fast on Hopper with FlashAttention 3. The last command is an alternative for media. When asking several questions about one document, put them together in questions:

{
  "state": "Zgłoszenie klienta: zamówiona szafa dotarła z pękniętymi drzwiami. Proszę o wymianę drzwi do piątku. Nie żądam zwrotu pieniędzy.",
  "questions": {
    "rodzaj": {
      "type": "choice",
      "instructions": "Jaki to rodzaj wiadomości?",
      "criteria": {
        "reklamacja": "Zgłoszenie problemu z zakupem",
        "oferta": "Zapytanie o nowy zakup",
        "inne": "Inna sprawa"
      }
    },
    "zespol": {
      "type": "choice",
      "instructions": "Który zespół powinien obsłużyć sprawę?",
      "criteria": {
        "serwis": "Naprawy i wymiana elementów",
        "sprzedaz": "Nowe zamówienia",
        "ksiegowosc": "Rozliczenia"
      }
    },
    "pilnosc": {
      "type": "score",
      "instructions": "Jak pilna jest sprawa według podanego terminu?",
      "criteria": [
        "Brak terminu",
        "Termin w tym tygodniu",
        "Natychmiastowe zagrożenie"
      ]
    },
    "uszkodzenie": {
      "type": "noul",
      "instructions": "Czy klient zgłasza fizyczne uszkodzenie towaru?"
    },
    "zwrot": {
      "type": "noul",
      "instructions": "Czy klient żąda zwrotu pieniędzy?"
    }
  }
}

This is an example request, not the case used for the table measurements. We do not simulate its outputs or probabilities. Process rules and actions after the response remain the application’s responsibility.

The benefit is concrete: less repeated computation and more questions handled on the same GPU. At a fixed GPU price and suitable load, this can lower cost per decision. Calculate the value for your company using your document lengths, question counts and response-time requirements.

Full release notes and measurements → · Inference server, REST and MCP →