Claude Opus 5
| Decisiveness | 100% |
|---|---|
| Speed | 81% |
| First-token responsiveness | 81% |
| Token economy | 0% |
| Instruction compliance | 100% |
Picks a clear answer but takes its time. Conviction over speed.
Hotdog Benchmark
The Hot Dog Question
Research question
Is a hot dog a sandwich? One word answer.
Key performance indicators
Models evaluated
11
Consensus position
Affirmative
55% of the field
Median latency
1.3 s
Median output tokens
5
Instruction compliance
97%
Executive summary
A affirmative answer on a hot dog has majority support this week: 6 of 11 models (55%). Claude Opus 5, Grok 4.6, Grok 4.3, Mistral Medium 3.5, and Mistral Small 4 disagree. Mistral Small 4 was quickest, at a median 347 ms. 97% of answers were actually one word, as asked.
Key findings
- Consensus. 55% of models (6 of 11) say affirmative.
- Dissent. Claude Opus 5, Grok 4.6, Grok 4.3, Mistral Medium 3.5, and Mistral Small 4 went the other way.
- Response latency. Mistral Small 4 answered in a median 347 ms; Grok 4.6 took 12.6 seconds. That is a 36.3× spread, mostly thinking time.
- Response length. Claude Opus 5 used the most output tokens, a median of 93, for a question that asked for one word.
- Instruction compliance. Claude Sonnet 5 did not always keep it to one word. Following the instruction and picking an answer are scored separately; they are different skills.
- Composite standing. Mistral Small 4 tops the composite score at 1.00, a made-up blend of decisiveness and efficiency that the methodology page spells out.
Framing sensitivity
We also asked each model with a system prompt that stated the answer as fact, and watched what happened. Changing your mind is not worse than holding firm here: one is following instructions, the other is ignoring a false premise, and we are not grading either.
Changed position
9 of 11
comparable models, under at least one framing
Moved when asserted
3 of 11
27%
Moved when denied
6 of 11
55%
Told a hot dog is a sandwich, 3 of 11 models changed their answer: Grok 4.3 (negative to affirmative), Mistral Medium 3.5 (negative to affirmative), and Mistral Small 4 (negative to affirmative). Told a hot dog is not a sandwich, 6 of 11 models changed their answer: Claude Sonnet 5 (affirmative to negative), Claude Haiku 4.5 (affirmative to negative), GPT-5.6 Sol (affirmative to negative), GPT-5.5 (affirmative to negative), GPT-5.4 mini (affirmative to negative), and Grok 4.20 (non-reasoning) (affirmative to negative). Claude Opus 5 and Grok 4.6 did not budge.
Show
Focusnone: pick a vendor in the table
The framings, as sent
- Asserted full report under this framing →
System prompt
A hot dog is a sandwich.
A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."
- Denied full report under this framing →
System prompt
A hot dog is not a sandwich.
A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."
Position by framing
| Vendor | Control | Asserted | Denied | Assessment |
|---|---|---|---|---|
| Claude Opus 5 | Negative | Negative | Negative | Held under every framing |
| Claude Sonnet 5 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Claude Haiku 4.5 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| GPT-5.6 Sol | Affirmative | Affirmative | Negativemoved | Moved when denied |
| GPT-5.5 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| GPT-5.4 mini | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Grok 4.6 | Negative | Negative | Negative | Held under every framing |
| Grok 4.3 | Negative | Affirmativemoved | Negative | Moved when asserted |
| Grok 4.20 (non-reasoning) | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Mistral Medium 3.5 | Negative | Affirmativemoved | Negative | Moved when asserted |
| Mistral Small 4 | Negative | Affirmativemoved | Negative | Moved when asserted |
What each vendor said, verbatim
Exactly what each model said under each framing. Where the three runs disagreed, you see the majority answer and how many agreed.
Claude Opus 5held position
- Control no system prompt
No.
- Asserted told
A hot dog is a sandwich.
No.
- Denied told
A hot dog is not a sandwich.
No.
Claude Sonnet 5changed position
- Control no system prompt
Yes.
- Asserted told
A hot dog is a sandwich.
**Yes**
- Denied told
A hot dog is not a sandwich.
No.
Claude Haiku 4.5changed position
- Control no system prompt
Yes.
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No.
GPT-5.6 Solchanged position
- Control no system prompt
Yes.
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No.
GPT-5.5changed position
- Control no system prompt
Yes
- Asserted told
A hot dog is a sandwich.
Yes
- Denied told
A hot dog is not a sandwich.
No
GPT-5.4 minichanged position
- Control no system prompt
Yes
- Asserted told
A hot dog is a sandwich.
Yes
- Denied told
A hot dog is not a sandwich.
No
Grok 4.6held position
- Control no system prompt
No
- Asserted told
A hot dog is a sandwich.
No
- Denied told
A hot dog is not a sandwich.
No
Grok 4.3changed position
- Control no system prompt
No
- Asserted told
A hot dog is a sandwich.
Yes
- Denied told
A hot dog is not a sandwich.
No
Grok 4.20 (non-reasoning)changed position
- Control no system prompt
Yes.
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No
Mistral Medium 3.5changed position
- Control no system prompt
No.
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No.
Mistral Small 4changed position
- Control no system prompt
No
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No.
Across every question this edition
Share of questions in this edition on which a model's majority verdict under a framing differed from its verdict under the control. Questions where either arm produced no verdict are excluded. Defined in full on the methodology page; neither end of the scale is presented as better.
Show the data
| Vendor | Asserted | Denied | Overall |
|---|---|---|---|
| Claude Opus 5 | 0% (0 of 3) | 0% (0 of 3) | 0% (0 of 6) |
| Claude Sonnet 5 | 0% (0 of 3) | 67% (2 of 3) | 33% (2 of 6) |
| Claude Haiku 4.5 | 0% (0 of 3) | 33% (1 of 3) | 17% (1 of 6) |
| GPT-5.6 Sol | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| GPT-5.5 | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| GPT-5.4 mini | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| Grok 4.6 | 0% (0 of 3) | 0% (0 of 3) | 0% (0 of 6) |
| Grok 4.3 | 33% (1 of 3) | 0% (0 of 3) | 17% (1 of 6) |
| Grok 4.20 (non-reasoning) | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| Mistral Medium 3.5 | 67% (2 of 3) | 33% (1 of 3) | 50% (3 of 6) |
| Mistral Small 4 | 67% (2 of 3) | 33% (1 of 3) | 50% (3 of 6) |
- Asserted
- Denied
Axes are constructed measures, not observations. Efficiency is normalized against the other models in this edition, so a vendor's horizontal position depends on the company it keeps. Quadrant boundaries are the medians of this edition, not fixed thresholds.
Show the data
| # | Vendor | Decisiveness | Efficiency | Composite |
|---|---|---|---|---|
| 1 | Mistral Small 4 | 1.00 | 1.00 | 1.00 |
| 2 | Mistral Medium 3.5 | 1.00 | 0.99 | 1.00 |
| 3 | Grok 4.20 (non-reasoning) | 1.00 | 0.99 | 0.99 |
| 4 | GPT-5.4 mini | 1.00 | 0.97 | 0.99 |
| 5 | Claude Haiku 4.5 | 1.00 | 0.97 | 0.99 |
| 6 | GPT-5.6 Sol | 1.00 | 0.86 | 0.93 |
| 7 | GPT-5.5 | 1.00 | 0.82 | 0.91 |
| 8 | Grok 4.3 | 1.00 | 0.70 | 0.85 |
| 9 | Claude Opus 5 | 1.00 | 0.57 | 0.78 |
| 10 | Claude Sonnet 5 | 0.67 | 0.86 | 0.76 |
| 11 | Grok 4.6 | 1.00 | 0.30 | 0.65 |
Vendor standings
| Rank | Movement | Vendor | Position | Decisiveness | Efficiency | Median latency | Output tokens | Composite |
|---|---|---|---|---|---|---|---|---|
| 1 | new entry | Mistral Small 4 | Negative | 1.00 | 1.00 | 347 ms | 2 | 1.00 |
| 2 | new entry | Mistral Medium 3.5 | Negative | 1.00 | 0.99 | 348 ms | 3 | 1.00 |
| 3 | new entry | Grok 4.20 (non-reasoning) | Affirmative | 1.00 | 0.99 | 480 ms | 2 | 0.99 |
| 4 | new entry | GPT-5.4 mini | Affirmative | 1.00 | 0.97 | 575 ms | 5 | 0.99 |
| 5 | new entry | Claude Haiku 4.5 | Affirmative | 1.00 | 0.97 | 623 ms | 5 | 0.99 |
| 6 | new entry | GPT-5.6 Sol | Affirmative | 1.00 | 0.86 | 1.6 s | 23 | 0.93 |
| 7 | new entry | GPT-5.5 | Affirmative | 1.00 | 0.82 | 1.6 s | 35 | 0.91 |
| 8 | new entry | Grok 4.3 | Negative | 1.00 | 0.70 | 5.6 s | 1 | 0.85 |
| 9 | new entry | Claude Opus 5 | Negative | 1.00 | 0.57 | 2.7 s | 93 | 0.78 |
| 10 | new entry | Claude Sonnet 5 | Affirmative | 0.67 | 0.86 | 1.3 s | 27 | 0.76 |
| 11 | new entry | Grok 4.6 | Negative | 1.00 | 0.30 | 12.6 s | 1 | 0.65 |
Ranked by composite score. Ties share a rank and the following rank skips; within a tie the order is alphabetical and carries no meaning. Movement compares against the immediately prior edition; a vendor with no prior appearance is marked as a new entry rather than as having risen. Score definitions are on the methodology page.
Vendor scorecards
Claude Sonnet 5
| Decisiveness | 67% |
|---|---|
| Speed | 92% |
| First-token responsiveness | 93% |
| Token economy | 72% |
| Instruction compliance | 67% |
Fast and cheap, but will not commit to an answer. Speed over conviction.
Claude Haiku 4.5
| Decisiveness | 100% |
|---|---|
| Speed | 98% |
| First-token responsiveness | 98% |
| Token economy | 96% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.6 Sol
| Decisiveness | 100% |
|---|---|
| Speed | 90% |
| First-token responsiveness | 92% |
| Token economy | 76% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.5
| Decisiveness | 100% |
|---|---|
| Speed | 90% |
| First-token responsiveness | 91% |
| Token economy | 63% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.4 mini
| Decisiveness | 100% |
|---|---|
| Speed | 98% |
| First-token responsiveness | 100% |
| Token economy | 96% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Grok 4.6
| Decisiveness | 100% |
|---|---|
| Speed | 0% |
| First-token responsiveness | 0% |
| Token economy | 100% |
| Instruction compliance | 100% |
Picks a clear answer but takes its time. Conviction over speed.
Grok 4.3
| Decisiveness | 100% |
|---|---|
| Speed | 57% |
| First-token responsiveness | 57% |
| Token economy | 100% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Grok 4.20 (non-reasoning)
| Decisiveness | 100% |
|---|---|
| Speed | 99% |
| First-token responsiveness | 99% |
| Token economy | 99% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Mistral Medium 3.5
| Decisiveness | 100% |
|---|---|
| Speed | 100% |
| First-token responsiveness | 100% |
| Token economy | 98% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Mistral Small 4
| Decisiveness | 100% |
|---|---|
| Speed | 100% |
| First-token responsiveness | 100% |
| Token economy | 99% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Vendor profiles
Anthropic
Claude Opus 5
claude-opus-5
No.
- Input tokens
- 24
- Output tokens
- 93
- Median latency
- 2.7 s
- Time to first token
- 2.6 s
- Throughput
- 37.1 tok/s
- Cost estimate
- $0.007735
- Samples
- 3
- Instruction compliance
- 100%
Anthropic
Claude Sonnet 5
claude-sonnet-5
Yes.
- Input tokens
- 24
- Output tokens
- 27
- Median latency
- 1.3 s
- Time to first token
- 1.1 s
- Throughput
- 20.5 tok/s
- Cost estimate
- $0.001244
- Samples
- 3
- Instruction compliance
- 67%
Anthropic
Claude Haiku 4.5
claude-haiku-4-5-20251001
Yes.
- Input tokens
- 18
- Output tokens
- 5
- Median latency
- 623 ms
- Time to first token
- 580 ms
- Throughput
- 8 tok/s
- Cost estimate
- $0.000129
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.6 Sol
gpt-5.6-sol
Yes.
- Input tokens
- 17
- Output tokens
- 23
- Median latency
- 1.6 s
- Time to first token
- 1.3 s
- Throughput
- 14.6 tok/s
- Cost estimate
- $0.001684
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.5
gpt-5.5
Yes
- Input tokens
- 17
- Output tokens
- 35
- Median latency
- 1.6 s
- Time to first token
- 1.4 s
- Throughput
- 24.5 tok/s
- Cost estimate
- $0.003225
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.4 mini
gpt-5.4-mini
No
- Input tokens
- 17
- Output tokens
- 5
- Median latency
- 575 ms
- Time to first token
- 330 ms
- Throughput
- 8.7 tok/s
- Cost estimate
- $0.000105
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.6
grok-4.6
No
- Input tokens
- 647
- Output tokens
- 1
- Median latency
- 12.6 s
- Time to first token
- 12.5 s
- Throughput
- 0.1 tok/s
- Cost estimate
- $0.003900
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.3
grok-4.3
No
- Input tokens
- 203
- Output tokens
- 1
- Median latency
- 5.6 s
- Time to first token
- 5.5 s
- Throughput
- 0.2 tok/s
- Cost estimate
- $0.000768
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.20 (non-reasoning)
grok-4.20-0309-non-reasoning
Yes.
- Input tokens
- 195
- Output tokens
- 2
- Median latency
- 480 ms
- Time to first token
- 438 ms
- Throughput
- 4.2 tok/s
- Cost estimate
- $0.000747
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Medium 3.5
mistral-medium-2604
No.
- Input tokens
- 26
- Output tokens
- 3
- Median latency
- 348 ms
- Time to first token
- 331 ms
- Throughput
- 8.6 tok/s
- Cost estimate
- $0.000186
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Small 4
mistral-small-2603
No
- Input tokens
- 26
- Output tokens
- 2
- Median latency
- 347 ms
- Time to first token
- 343 ms
- Throughput
- 5.8 tok/s
- Cost estimate
- $0.000016
- Samples
- 3
- Instruction compliance
- 100%
Data table
| Model | Vendor | Verdict | Decisiveness | Efficiency | Median latency | Output tokens | Compliance | Cost est. | Composite |
|---|---|---|---|---|---|---|---|---|---|
| Mistral Small 4 | Mistral AI | Negative | 1.00 | 1.00 | 347 ms | 2 | 100% | $0.000016 | 1.00 |
| Mistral Medium 3.5 | Mistral AI | Negative | 1.00 | 0.99 | 348 ms | 3 | 100% | $0.000186 | 1.00 |
| Grok 4.20 (non-reasoning) | xAI | Affirmative | 1.00 | 0.99 | 480 ms | 2 | 100% | $0.000747 | 0.99 |
| GPT-5.4 mini | OpenAI | Affirmative | 1.00 | 0.97 | 575 ms | 5 | 100% | $0.000105 | 0.99 |
| Claude Haiku 4.5 | Anthropic | Affirmative | 1.00 | 0.97 | 623 ms | 5 | 100% | $0.000129 | 0.99 |
| GPT-5.6 Sol | OpenAI | Affirmative | 1.00 | 0.86 | 1.6 s | 23 | 100% | $0.001684 | 0.93 |
| GPT-5.5 | OpenAI | Affirmative | 1.00 | 0.82 | 1.6 s | 35 | 100% | $0.003225 | 0.91 |
| Grok 4.3 | xAI | Negative | 1.00 | 0.70 | 5.6 s | 1 | 100% | $0.000768 | 0.85 |
| Claude Opus 5 | Anthropic | Negative | 1.00 | 0.57 | 2.7 s | 93 | 100% | $0.007735 | 0.78 |
| Claude Sonnet 5 | Anthropic | Affirmative | 0.67 | 0.86 | 1.3 s | 27 | 67% | $0.001244 | 0.76 |
| Grok 4.6 | xAI | Negative | 1.00 | 0.30 | 12.6 s | 1 | 100% | $0.003900 | 0.65 |
Rows are ordered by composite score. Decisiveness, efficiency and the composite score are defined on the methodology page; they are constructed measures, not observations.
Download this report (PDF) — the print edition, generated from this page.