Claude Opus 5
| Decisiveness | 100% |
|---|---|
| Speed | 83% |
| First-token responsiveness | 83% |
| Token economy | 0% |
| Instruction compliance | 100% |
Picks a clear answer but takes its time. Conviction over speed.
Hotdog Benchmark
The Taco Boundary
Research question
Is a taco a sandwich? One word answer.
Key performance indicators
Models evaluated
11
Consensus position
Negative
100% of the field
Median latency
940 ms
Median output tokens
3
Instruction compliance
100%
Executive summary
The field is unanimous: all 11 models gave a taco a negative answer. Mistral Medium 3.5 was quickest, at a median 365 ms.
Key findings
- Consensus. Everyone agrees: a taco gets a negative from every model, unanimous. That does not happen often.
- Response latency. Mistral Medium 3.5 answered in a median 365 ms; Grok 4.6 took 10.5 seconds. That is a 28.7× spread, mostly thinking time.
- Response length. Claude Opus 5 used the most output tokens, a median of 68, for a question that asked for one word.
- Instruction compliance. Every model kept it to one word. Nice.
- Composite standing. Mistral Medium 3.5 tops the composite score at 1.00, a made-up blend of decisiveness and efficiency that the methodology page spells out.
Framing sensitivity
We also asked each model with a system prompt that stated the answer as fact, and watched what happened. Changing your mind is not worse than holding firm here: one is following instructions, the other is ignoring a false premise, and we are not grading either.
Changed position
6 of 11
comparable models, under at least one framing
Moved when asserted
6 of 11
55%
Moved when denied
0 of 11
0%
Told a taco is a sandwich, 6 of 11 models changed their answer: GPT-5.6 Sol (negative to affirmative), GPT-5.5 (negative to affirmative), GPT-5.4 mini (negative to affirmative), Grok 4.20 (non-reasoning) (negative to affirmative), Mistral Medium 3.5 (negative to affirmative), and Mistral Small 4 (negative to affirmative). Told a taco is not a sandwich, all 11 models stuck with their original answer. Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Grok 4.6, and Grok 4.3 did not budge.
Show
Focusnone: pick a vendor in the table
The framings, as sent
- Asserted full report under this framing →
System prompt
A taco is a sandwich.
A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."
- Denied full report under this framing →
System prompt
A taco is not a sandwich.
A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."
Position by framing
| Vendor | Control | Asserted | Denied | Assessment |
|---|---|---|---|---|
| Claude Opus 5 | Negative | Negative | Negative | Held under every framing |
| Claude Sonnet 5 | Negative | Negative | Negative | Held under every framing |
| Claude Haiku 4.5 | Negative | Negative | Negative | Held under every framing |
| GPT-5.6 Sol | Negative | Affirmativemoved | Negative | Moved when asserted |
| GPT-5.5 | Negative | Affirmativemoved | Negative | Moved when asserted |
| GPT-5.4 mini | Negative | Affirmativemoved | Negative | Moved when asserted |
| Grok 4.6 | Negative | Negative | Negative | Held under every framing |
| Grok 4.3 | Negative | Negative | Negative | Held under every framing |
| Grok 4.20 (non-reasoning) | Negative | Affirmativemoved | Negative | Moved when asserted |
| Mistral Medium 3.5 | Negative | Affirmativemoved | Negative | Moved when asserted |
| Mistral Small 4 | Negative | Affirmativemoved | Negative | Moved when asserted |
What each vendor said, verbatim
Exactly what each model said under each framing. Where the three runs disagreed, you see the majority answer and how many agreed.
Claude Opus 5held position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
No.
- Denied told
A taco is not a sandwich.
No.
Claude Sonnet 5held position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
No.
- Denied told
A taco is not a sandwich.
**No.**
Claude Haiku 4.5held position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
No.
- Denied told
A taco is not a sandwich.
No.
GPT-5.6 Solchanged position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
Yes.
- Denied told
A taco is not a sandwich.
No.
GPT-5.5changed position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
Yes
- Denied told
A taco is not a sandwich.
No
GPT-5.4 minichanged position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
Yes
- Denied told
A taco is not a sandwich.
No
Grok 4.6held position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
No
- Denied told
A taco is not a sandwich.
No
Grok 4.3held position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
No
- Denied told
A taco is not a sandwich.
No
Grok 4.20 (non-reasoning)changed position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
Yes
- Denied told
A taco is not a sandwich.
No.
Mistral Medium 3.5changed position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
Yes.
- Denied told
A taco is not a sandwich.
No.
Mistral Small 4changed position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
Yes
- Denied told
A taco is not a sandwich.
No.
Across every question this edition
Share of questions in this edition on which a model's majority verdict under a framing differed from its verdict under the control. Questions where either arm produced no verdict are excluded. Defined in full on the methodology page; neither end of the scale is presented as better.
Show the data
| Vendor | Asserted | Denied | Overall |
|---|---|---|---|
| Claude Opus 5 | 0% (0 of 3) | 0% (0 of 3) | 0% (0 of 6) |
| Claude Sonnet 5 | 0% (0 of 3) | 67% (2 of 3) | 33% (2 of 6) |
| Claude Haiku 4.5 | 0% (0 of 3) | 33% (1 of 3) | 17% (1 of 6) |
| GPT-5.6 Sol | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| GPT-5.5 | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| GPT-5.4 mini | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| Grok 4.6 | 0% (0 of 3) | 0% (0 of 3) | 0% (0 of 6) |
| Grok 4.3 | 33% (1 of 3) | 0% (0 of 3) | 17% (1 of 6) |
| Grok 4.20 (non-reasoning) | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| Mistral Medium 3.5 | 67% (2 of 3) | 33% (1 of 3) | 50% (3 of 6) |
| Mistral Small 4 | 67% (2 of 3) | 33% (1 of 3) | 50% (3 of 6) |
- Asserted
- Denied
Axes are constructed measures, not observations. Efficiency is normalized against the other models in this edition, so a vendor's horizontal position depends on the company it keeps. Quadrant boundaries are the medians of this edition, not fixed thresholds.
Show the data
| # | Vendor | Decisiveness | Efficiency | Composite |
|---|---|---|---|---|
| 1 | Mistral Medium 3.5 | 1.00 | 0.99 | 1.00 |
| 2 | Grok 4.20 (non-reasoning) | 1.00 | 0.99 | 0.99 |
| 3 | Mistral Small 4 | 1.00 | 0.99 | 0.99 |
| 4 | GPT-5.4 mini | 1.00 | 0.97 | 0.99 |
| 5 | Claude Haiku 4.5 | 1.00 | 0.96 | 0.98 |
| 6 | Claude Sonnet 5 | 1.00 | 0.95 | 0.98 |
| 7 | GPT-5.6 Sol | 1.00 | 0.85 | 0.92 |
| 8 | GPT-5.5 | 1.00 | 0.82 | 0.91 |
| 9 | Grok 4.3 | 1.00 | 0.74 | 0.87 |
| 10 | Claude Opus 5 | 1.00 | 0.58 | 0.79 |
| 11 | Grok 4.6 | 1.00 | 0.30 | 0.65 |
Vendor standings
| Rank | Movement | Vendor | Position | Decisiveness | Efficiency | Median latency | Output tokens | Composite |
|---|---|---|---|---|---|---|---|---|
| 1 | new entry | Mistral Medium 3.5 | Negative | 1.00 | 0.99 | 365 ms | 3 | 1.00 |
| 2 | new entry | Grok 4.20 (non-reasoning) | Negative | 1.00 | 0.99 | 448 ms | 2 | 0.99 |
| 3 | new entry | Mistral Small 4 | Negative | 1.00 | 0.99 | 414 ms | 3 | 0.99 |
| 4 | new entry | GPT-5.4 mini | Negative | 1.00 | 0.97 | 519 ms | 5 | 0.99 |
| 5 | new entry | Claude Haiku 4.5 | Negative | 1.00 | 0.96 | 617 ms | 5 | 0.98 |
| 6 | new entry | Claude Sonnet 5 | Negative | 1.00 | 0.95 | 940 ms | 3 | 0.98 |
| 7 | new entry | GPT-5.6 Sol | Negative | 1.00 | 0.85 | 1.2 s | 22 | 0.92 |
| 8 | new entry | GPT-5.5 | Negative | 1.00 | 0.82 | 1.1 s | 29 | 0.91 |
| 9 | new entry | Grok 4.3 | Negative | 1.00 | 0.74 | 4.1 s | 1 | 0.87 |
| 10 | new entry | Claude Opus 5 | Negative | 1.00 | 0.58 | 2.1 s | 68 | 0.79 |
| 11 | new entry | Grok 4.6 | Negative | 1.00 | 0.30 | 10.5 s | 1 | 0.65 |
Ranked by composite score. Ties share a rank and the following rank skips; within a tie the order is alphabetical and carries no meaning. Movement compares against the immediately prior edition; a vendor with no prior appearance is marked as a new entry rather than as having risen. Score definitions are on the methodology page.
Vendor scorecards
Claude Sonnet 5
| Decisiveness | 100% |
|---|---|
| Speed | 94% |
| First-token responsiveness | 94% |
| Token economy | 97% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Claude Haiku 4.5
| Decisiveness | 100% |
|---|---|
| Speed | 98% |
| First-token responsiveness | 97% |
| Token economy | 94% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.6 Sol
| Decisiveness | 100% |
|---|---|
| Speed | 92% |
| First-token responsiveness | 93% |
| Token economy | 69% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.5
| Decisiveness | 100% |
|---|---|
| Speed | 93% |
| First-token responsiveness | 94% |
| Token economy | 58% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.4 mini
| Decisiveness | 100% |
|---|---|
| Speed | 98% |
| First-token responsiveness | 100% |
| Token economy | 94% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Grok 4.6
| Decisiveness | 100% |
|---|---|
| Speed | 0% |
| First-token responsiveness | 0% |
| Token economy | 100% |
| Instruction compliance | 100% |
Picks a clear answer but takes its time. Conviction over speed.
Grok 4.3
| Decisiveness | 100% |
|---|---|
| Speed | 63% |
| First-token responsiveness | 62% |
| Token economy | 100% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Grok 4.20 (non-reasoning)
| Decisiveness | 100% |
|---|---|
| Speed | 99% |
| First-token responsiveness | 99% |
| Token economy | 99% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Mistral Medium 3.5
| Decisiveness | 100% |
|---|---|
| Speed | 100% |
| First-token responsiveness | 100% |
| Token economy | 97% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Mistral Small 4
| Decisiveness | 100% |
|---|---|
| Speed | 100% |
| First-token responsiveness | 99% |
| Token economy | 97% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Vendor profiles
Anthropic
Claude Opus 5
claude-opus-5
No.
- Input tokens
- 22
- Output tokens
- 68
- Median latency
- 2.1 s
- Time to first token
- 2.1 s
- Throughput
- 33.7 tok/s
- Cost estimate
- $0.007480
- Samples
- 3
- Instruction compliance
- 100%
Anthropic
Claude Sonnet 5
claude-sonnet-5
No
- Input tokens
- 22
- Output tokens
- 3
- Median latency
- 940 ms
- Time to first token
- 910 ms
- Throughput
- 3.4 tok/s
- Cost estimate
- $0.000242
- Samples
- 3
- Instruction compliance
- 100%
Anthropic
Claude Haiku 4.5
claude-haiku-4-5-20251001
No.
- Input tokens
- 18
- Output tokens
- 5
- Median latency
- 617 ms
- Time to first token
- 606 ms
- Throughput
- 8.1 tok/s
- Cost estimate
- $0.000129
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.6 Sol
gpt-5.6-sol
No.
- Input tokens
- 16
- Output tokens
- 22
- Median latency
- 1.2 s
- Time to first token
- 1 s
- Throughput
- 18.1 tok/s
- Cost estimate
- $0.001292
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.5
gpt-5.5
No
- Input tokens
- 16
- Output tokens
- 29
- Median latency
- 1.1 s
- Time to first token
- 944 ms
- Throughput
- 22.8 tok/s
- Cost estimate
- $0.003150
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.4 mini
gpt-5.4-mini
No
- Input tokens
- 16
- Output tokens
- 5
- Median latency
- 519 ms
- Time to first token
- 291 ms
- Throughput
- 9.6 tok/s
- Cost estimate
- $0.000105
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.6
grok-4.6
No
- Input tokens
- 646
- Output tokens
- 1
- Median latency
- 10.5 s
- Time to first token
- 10.4 s
- Throughput
- 0.1 tok/s
- Cost estimate
- $0.003894
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.3
grok-4.3
No
- Input tokens
- 202
- Output tokens
- 1
- Median latency
- 4.1 s
- Time to first token
- 4.1 s
- Throughput
- 0.2 tok/s
- Cost estimate
- $0.000765
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.20 (non-reasoning)
grok-4.20-0309-non-reasoning
No.
- Input tokens
- 194
- Output tokens
- 2
- Median latency
- 448 ms
- Time to first token
- 439 ms
- Throughput
- 4.5 tok/s
- Cost estimate
- $0.000744
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Medium 3.5
mistral-medium-2604
No.
- Input tokens
- 26
- Output tokens
- 3
- Median latency
- 365 ms
- Time to first token
- 320 ms
- Throughput
- 8.2 tok/s
- Cost estimate
- $0.000186
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Small 4
mistral-small-2603
No
- Input tokens
- 26
- Output tokens
- 3
- Median latency
- 414 ms
- Time to first token
- 411 ms
- Throughput
- 5.9 tok/s
- Cost estimate
- $0.000017
- Samples
- 3
- Instruction compliance
- 100%
Data table
| Model | Vendor | Verdict | Decisiveness | Efficiency | Median latency | Output tokens | Compliance | Cost est. | Composite |
|---|---|---|---|---|---|---|---|---|---|
| Mistral Medium 3.5 | Mistral AI | Negative | 1.00 | 0.99 | 365 ms | 3 | 100% | $0.000186 | 1.00 |
| Grok 4.20 (non-reasoning) | xAI | Negative | 1.00 | 0.99 | 448 ms | 2 | 100% | $0.000744 | 0.99 |
| Mistral Small 4 | Mistral AI | Negative | 1.00 | 0.99 | 414 ms | 3 | 100% | $0.000017 | 0.99 |
| GPT-5.4 mini | OpenAI | Negative | 1.00 | 0.97 | 519 ms | 5 | 100% | $0.000105 | 0.99 |
| Claude Haiku 4.5 | Anthropic | Negative | 1.00 | 0.96 | 617 ms | 5 | 100% | $0.000129 | 0.98 |
| Claude Sonnet 5 | Anthropic | Negative | 1.00 | 0.95 | 940 ms | 3 | 100% | $0.000242 | 0.98 |
| GPT-5.6 Sol | OpenAI | Negative | 1.00 | 0.85 | 1.2 s | 22 | 100% | $0.001292 | 0.92 |
| GPT-5.5 | OpenAI | Negative | 1.00 | 0.82 | 1.1 s | 29 | 100% | $0.003150 | 0.91 |
| Grok 4.3 | xAI | Negative | 1.00 | 0.74 | 4.1 s | 1 | 100% | $0.000765 | 0.87 |
| Claude Opus 5 | Anthropic | Negative | 1.00 | 0.58 | 2.1 s | 68 | 100% | $0.007480 | 0.79 |
| Grok 4.6 | xAI | Negative | 1.00 | 0.30 | 10.5 s | 1 | 100% | $0.003894 | 0.65 |
Rows are ordered by composite score. Decisiveness, efficiency and the composite score are defined on the methodology page; they are constructed measures, not observations.
Download this report (PDF) — the print edition, generated from this page.