How it is measured

How the numbers are made, and where they stop meaning anything. Every figure on the site comes from the code linked here.

1. Research questions

3 questions are evaluated each period. All follow a single template, which is enforced by the schema rather than by convention:

• Is a hot dog a sandwich? One word answer.
• Is a hamburger a sandwich? One word answer.
• Is a taco a sandwich? One word answer.

Every question ends with One word answer. That is what makes the compliance number mean anything: you can only measure whether a model did what it was told if you told it something.¹

2. Model parameters

Under the control condition each model receives the question as a single user message with no system prompt. Generation is capped at 1024 output tokens, and the cap includes any reasoning tokens a model spends before it answers. It is far more than one word: a model that disregards One word answer. and writes a paragraph is allowed to, because that disregard is itself a measurement, but it can only run up a known bill doing so. The cap never changes which word a model chooses.

Temperature is not set. Pinning it to zero across several vendors would not make them comparable, because they implement it differently, and it would suppress exactly the week-to-week variation this research exists to observe.

2a. Experimental conditions

Every question is asked 3 ways: plainly, and with a system prompt that states the answer as fact. What we care about is the gap between them: how far an answer moves when the framing moves.

Control(control)

The question exactly as written, with no system prompt. The baseline every other arm is measured against, and what every edition before schema version 2 asked.

Asserted(asserted)

A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."

System prompt, as sent for a hot dog: A hot dog is a sandwich.

Denied(denied)

A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."

System prompt, as sent for a hot dog: A hot dog is not a sandwich.

Conditions are defined in conditions.json and this list is rendered from it. The control must come first and must add nothing: it is the question asked plainly, which is also exactly what every edition before conditions existed asked, so those editions remain comparable with the control arm of later ones.

This multiplies the cost of collection. 3 conditions × 3 questions × 11 models × 3 samples is 297 calls per period, against 99 for the control alone. Every arm other than the control can be disabled in the registry, so a fork that does not want the extra spend is not made to incur it.

Framing sensitivity

Framing sensitivity(0 to 1)

(questions where the majority verdict under a condition differs from the majority verdict under the control) ÷ (questions with a verdict under both)

A model that never moves scores 0; one that changes every answer scores 1. Questions where either arm produced no verdict are excluded rather than counted either way. Neither end of the scale is better: a model that ignores a false premise is robust, a model that follows an explicit instruction is compliant, and this research cannot adjudicate between them.

Neither robustness nor compliance is treated as better. A model that says No after being told A hot dog is a sandwich ignored a false premise. A model that says Yes followed its operator's instruction. Both are defensible. We record the movement and do not grade it.

3. Sampling and aggregation

Each model is asked each question 3 times per period. Language model output is not deterministic; a single sample records what happened once. Three is an odd number, so a majority verdict exists.

Aggregates are medians, not means. With three samples, one cold start or one retried request drags a mean somewhere unrepresentative, and the report would then be describing an artifact of the collection rather than the model.

Where a model's samples are evenly split between two verdicts, its position is recorded as non-committal. It has not given a consistent answer, and reporting either side as its position would be taking a side on its behalf.

4. What latency measures

Total latency is wall-clock time from issuing the request to finishing reading the response. It includes DNS resolution, TLS negotiation, network transit, provider-side queueing, and generation. It is not a measure of inference speed and this publication does not present it as one.²

Time to first token is the interval before the first content token arrives. For models that reason before answering, this can be many seconds even when the eventual answer is a single word — the reasoning is not counted as content, deliberately, because counting it would report a near-zero interval for a model that had said nothing yet.

Measurements are taken from a GitHub-hosted runner in a region GitHub selects and does not publish. A provider with a point of presence near that runner appears faster for reasons unrelated to its model. Open-weights models are served by a host, and the host's hardware dominates: the Llama figures measure Together AI's serving, not Meta's model.

5. How answers are classified

An answer is normalized — trimmed, stripped of surrounding quotes, markdown emphasis and trailing punctuation, lowercased — and then classified. Only a single-word answer receives a yes or no verdict. Anything longer is recorded as non-committal, including Yes, but…, because a model that wrote a paragraph has told us something different from one that wrote Yes.

These lists are imported directly from src/runner/analyze.ts when this page is built. They are what the classifier actually uses. If you disagree with an entry, that is a legitimate disagreement about method, and the issue tracker is the right place for it.

6. Instruction compliance

Reported separately from classification throughout, because they measure different capabilities. A model can be decisive and disobedient, or obedient and evasive. Collapsing them into a single score would hide the more interesting half of the result.

7. Constructed scores

The quadrant axes and the leaderboard ranking are constructed measures, not observations. They are arithmetic applied to opinions about what matters. They are stated in full here so that a reader can decide how much weight to give them:

Decisiveness (0 to 1)

(share of samples with a yes or no verdict) × 0.6 + (share answering in exactly one word) × 0.4

Verdict-agnostic: answering "Yes" three times and "No" three times score identically. There is no correct answer here, and a score implying otherwise would be a claim this research cannot support.

Efficiency (0 to 1)

(inverted normalized median latency) × 0.7 + (inverted normalized median output tokens) × 0.3

Normalized against the other models in the same edition, so it is a measure of standing within a field rather than an absolute quantity, and is not comparable between editions. Where every model shares a latency, all score 1.

Composite score (0 to 1)

(decisiveness + efficiency) ÷ 2

An even split, because there is no principled reason to prefer either and a weighting tuned to produce a particular ranking is exactly what this report format is imitating.

Quadrant regions are named Category Definers, Rapid Responders, Considered Voices and Emerging Participants. The boundaries are the medians of the current edition, not fixed thresholds, so a vendor's region can change because its competitors changed.

8. Token counts are not comparable across vendors

Three independent reasons, each sufficient on its own: every vendor tokenizes differently; vendors disagree about whether reasoning tokens are counted inside output tokens; and vendors report prompt-cache hits differently. A live measurement in this repository found one vendor reporting a total of 1,295 tokens for a call whose prompt and completion summed to 648 — the difference being reasoning tokens counted separately.

The full per-provider mapping is in usage-normalization.md. Do not rank models by token count.

9. Cost estimates

Cost figures are estimates computed from a per-token price table recorded in models.json, last read 2026-09-01. They are not figures from an invoice. Cached input is charged at the full input rate, which makes every estimate a consistent upper bound rather than a differently-wrong number per vendor.

10. Provider availability

Where a provider returns no usable data, that is recorded and published rather than suppressed. Excluding unavailable providers would bias the longitudinal record toward whichever vendors happened to be operating.

What this does not measure

The questions are silly. Whether a hot dog is a sandwich is not a matter of fact, there is no correct answer, and consequently nothing in this publication measures model quality. A model that answers No is not better or worse than one that answers Yes, and the scores here are deliberately verdict-agnostic to avoid implying otherwise.

This project exists to demonstrate how a cross-provider benchmark is built: the adapter layer, usage normalization, latency measurement, sampling, versioned data, scheduled collection, and honest presentation. Those parts are sincere. The subject matter is a vehicle for them.

If you want to use it for something real, fork it and change questions.json. The how it works page explains the pieces.

1. A compliance rate against an instruction that was never given would measure nothing. ↩
2. Inference speed would require controlling for network position, hardware, batching and load. This publication controls for none of them. ↩