basal-omni takes the top spot on Benchmark Heaven’s open-weights leaderboard

What first place means

A decision model receives context, a question and the allowed answers. Its job is to select the right option quickly and return useful probabilities. Both decision quality and confidence matter: the latter helps determine when to trust automation and when to request a review.

Capability Score averages two axes: Intelligence — decision quality adjusted for factors including chance performance — and Calibration, which evaluates probabilities. Cost and latency determine eligibility: at most twice the Jev reference values, or roughly USD 0.0646 per 1,000 decisions and an adjusted median latency of 1.23 s.

The score of 82.54 is an index value, not a percentage of correct answers. Likewise, 91.02 on the Calibration axis does not mean “91.02% accuracy”. JevBench methodology.

Model comparison

Below are the top five open-weights models by Capability, plus two reference points: H2O-Lightning and Jev. All values come from Benchmark Heaven’s public export; higher is better in each of the three score columns.

Jev 1.13.0 is a closed API model shown as a reference, without an open-weights rank. We do not substitute results from newer Jev versions. Download the comparison snapshot and sources (JSON) ↓

ultra leads Quyet-1.0-Large by 0.85 points, Jeff-1.0-Large by 1.03 points and the Jev 1.13.0 reference by 5.41 Capability points. Jeff has a slightly higher calibration score; basal leads through the combination of calibration and higher Intelligence. First place is a combined result, rather than a win in every column.

Benchmark Heaven also publishes a Composite Score, incorporating speed and cost. In its default option A, H2O-Lightning-4B v1.1 leads the open-weights board with 72.52. basal-omni-1.0-ultra is second with 71.76. This is a useful distinction: a 4B model may be a better fit for simpler, high-volume traffic, while ultra delivers the highest Capability within the required time and cost budget.

Microsoft, OpenAI and Perplexity

A Polish model with open weights achieved a higher Capability Score than OpenAI Decisions and Microsoft-Decision-1. The margins are 9.04 and 11.75 points respectively. This comparison covers the specific decision systems, versions and configurations published by Benchmark Heaven.

Source: JevBench v1.6.28 export. * Self-hosted model costs are estimates under the benchmark methodology; API costs use published tariffs and test usage. These figures do not compare GPU rental bills.

Perplexity needs a separate explanation. Decider v1.1 27B has a slightly higher Capability score: 82.79 versus basal’s 82.54. Its estimated cost is USD 0.21733 per 1,000 decisions — about 4.9 times ultra’s and above the official USD 0.0646 cap. It therefore does not qualify for the headline Capability ranking under the default caps. basal achieves a similar index score at a substantially lower estimated cost.

In the combined Capability ranking of eligible open-weights models and APIs, ultra takes second place. Sqwish’s kettei-fat leads with 86.31. This puts basal’s result in context: first among open weights and ahead of OpenAI’s and Microsoft’s specialised decision systems in this evaluation.

For Microsoft, we use the officially ranked measurement of 70.78. The board also contains a preliminary A2 re-run scoring 72.19, excluded from rankings. That score is also below ultra’s. API results and measurement status →

How it was measured

According to the v1.6.28 release report, the Benchmark Heaven team ran ultra’s public weights on one H200 141 GB GPU using our unmodified basal 2.1.0 engine. The evaluation contained 1,500 decisions: 300 public and 1,200 sealed. This assessment was conducted outside the basal project.

The model answered 1,485 questions. The 15 longest inputs exceeded available memory; refusals counted as incorrect answers, keeping the full denominator of 1,500. Measured median latency was about 102 ms, locally on the GPU. The board applies an adjustment of 2 × time + 150 ms, giving roughly 354 ms. The figure of USD 0.04465 per 1,000 decisions is a benchmark-methodology estimate, not a basal service price.

The board combines evaluations from different question draws. In the saved JSON export, Ultra’s draw is marked as awaiting anchor-model runs used to equate difficulty across draws. First place describes the published ordering; we do not present small point differences as proof of statistical superiority on every task. This leaderboard evaluates text decisions — image and audio quality require separate tests.

Measurement conditions, refusals and versions: JevBench v1.6.28 report. Current board scores and ranks: public JSON export.

What this means for users

Behind the result is a model you can download and run on your own infrastructure. basal-omni-1.0-ultra has 31 billion parameters, is based on Google Gemma 4 and releases its weights under Apache-2.0. It accepts text and images; the omni family’s mini and max variants handle audio. Strong text performance is one part of a broader tool.

In a business, a model like this can assess whether an answer is supported by a source, select a tool for an agent or choose how to handle a document. The application defines the question and allowed answers; the model returns a decision with probabilities. Local deployment lets case contents remain within the organisation’s infrastructure.

The result is a reason to test the model on your own workflow. Model selection, automation thresholds and error costs should still be grounded in labelled examples from your business. Our goal is to combine model quality with a practical engine: REST API, MCP, and decision and task schemas that integrate with existing software.

Try basal

This result is an important step for an open decision-model ecosystem developed in Poland. Thank you to the community for testing, issue reports and engine contributions, and to the Benchmark Heaven team for the independent evaluation and transparent reporting.

• Download basal-omni-1.0-ultra and read its model card.
• Explore the basal engine, REST API and MCP.
• Watch recorded use cases in Basal Lab.
• Compare models on our Werdykt benchmark as well.