After a run, every item has a Bradley-Terry score. Sort the items by that score, and for each adjacent pair take the probability the fitted model gives the higher item of beating the one directly below it. Consistency is the mean of those probabilities across every pair in every question. It runs from 50%, where every neighbour is a coin flip, to 100%, where every neighbour is cleanly separated.
Only adjacent pairs are counted, but each one carries far more than a single head-to-head. The gap between two neighbours is a difference of Bradley-Terry scores, and a Bradley-Terry score is fitted to every comparison that item took part in. So a neighbouring pair’s separation already reflects how both items fared against the rest of the field, not just against each other.
Higher is better, because a judge that separates items decisively needs far fewer comparisons to reach a confident ranking. But it is only half the story.
Consistency says nothing about whether the ranking is right. A judge that always picks whichever answer comes first alphabetically is perfectly consistent: every neighbour is separated with total confidence, and the ranking is useless. Think of it the way precision is separated from accuracy on a target. Consensus Match is where the darts land on average. Consistency is how tightly they group.
Language models tend to favour whichever option they see first. NanoJudge measures that tendency across each run and corrects for it, so it does not distort the ranking. Positional Bias reports how strong the tendency was before correction, as the split between the two positions averaged across questions: the first number is how often the judge picked the item it was shown first, the second how often it picked the item shown second. 50/50 is the ideal, and means the judge showed no preference for either position at all. A judge at 68/32 leaned on the first position and one at 35/65 on the second, and the further from 50/50 in either direction, the more of every comparison the judge spent reacting to the order it saw the items in rather than the items themselves. The correction can only recover so much of that.
For each question the model produces a ranking, and Consensus Match measures how closely that ranking agrees with the consensus ranking, averaged across all questions. Higher is better.
Most benchmark questions have no external ground truth, so the benchmark uses the consensus of the panel as a proxy for correct. Pool every other model’s judgements on a question into one Bradley-Terry fit, and treat that ranking as the answer. Each model is scored against the consensus of the others, never against a consensus that includes its own verdicts. The next section explains why.
The wisdom of crowds is the observation that a group of independent, imperfect guessers is often more accurate than any single expert among them. It works because each guesser’s errors point in a different direction, so they cancel when combined, while the true answer is the one thing they all lean toward. The same thing happens here. The panel’s models were built by different labs on different data, so where they disagree their mistakes tend to cancel, and where they agree is the best estimate of the answer available.
That is why consensus is the main score. In computer chess, taking the average of two similarly rated chess AIs (eg Stockfish and Lc0) outperforms using either one on its own, because they cover each other’s blind spots. The different architectures and training data of LLMs means that a group of them is likely stronger than any one on its own.
It is not perfect. A weak model pulls the consensus toward its own mistakes, and its vote counts as much as the best model’s. If several models share a bias, such as favouring longer or more confident-sounding answers, the consensus inherits it, and a model that correctly resists that bias is marked down for disagreeing.
It is still the best thing we have. As the panel grows and the models improve, the consensus tightens toward the truth. And its failures are visible rather than hidden: a model far from consensus with high consistency is either seeing something the others miss or confidently wrong, and the question browser lets you read the answers and decide which.
In the future, the consensus will be based around a stronger panel: the most capable model from each of the leading AI labs. Every model on the leaderboard will then be measured against the strongest judges available rather than against its peers, and smaller models will no longer affect the ground truth we base scores around.
Input and Output are the list prices per million tokens. Total Cost is what it actually cost to run this model through the entire benchmark: input price times input tokens used, plus output price times output tokens used. It reflects how verbose the model was, not just its price list.