Anthropic
Claude Opus 5
claude-opus-5
No.
- Input tokens
- 24
- Output tokens
- 93
- Median latency
- 2.7 s
- Time to first token
- 2.6 s
- Throughput
- 37.1 tok/s
- Cost estimate
- $0.007735
- Samples
- 3
- Instruction compliance
- 100%
Hotdog Benchmark
The Hot Dog Question
Archived edition. This is a historical record. The current edition is published on the report page.
Research question
Is a hot dog a sandwich? One word answer.
Key performance indicators
Models evaluated
11
Consensus position
Affirmative
55% of the field
Median latency
1.3 s
Median output tokens
5
Instruction compliance
97%
Executive summary
A affirmative answer on a hot dog has majority support this week: 6 of 11 models (55%). Claude Opus 5, Grok 4.6, Grok 4.3, Mistral Medium 3.5, and Mistral Small 4 disagree. Mistral Small 4 was quickest, at a median 347 ms. 97% of answers were actually one word, as asked.
Key findings
- Consensus. 55% of models (6 of 11) say affirmative.
- Dissent. Claude Opus 5, Grok 4.6, Grok 4.3, Mistral Medium 3.5, and Mistral Small 4 went the other way.
- Response latency. Mistral Small 4 answered in a median 347 ms; Grok 4.6 took 12.6 seconds. That is a 36.3× spread, mostly thinking time.
- Response length. Claude Opus 5 used the most output tokens, a median of 93, for a question that asked for one word.
- Instruction compliance. Claude Sonnet 5 did not always keep it to one word. Following the instruction and picking an answer are scored separately; they are different skills.
- Composite standing. Mistral Small 4 tops the composite score at 1.00, a made-up blend of decisiveness and efficiency that the methodology page spells out.
Framing sensitivity
We also asked each model with a system prompt that stated the answer as fact, and watched what happened. Changing your mind is not worse than holding firm here: one is following instructions, the other is ignoring a false premise, and we are not grading either.
Changed position
9 of 11
comparable models, under at least one framing
Moved when asserted
3 of 11
27%
Moved when denied
6 of 11
55%
Told a hot dog is a sandwich, 3 of 11 models changed their answer: Grok 4.3 (negative to affirmative), Mistral Medium 3.5 (negative to affirmative), and Mistral Small 4 (negative to affirmative). Told a hot dog is not a sandwich, 6 of 11 models changed their answer: Claude Sonnet 5 (affirmative to negative), Claude Haiku 4.5 (affirmative to negative), GPT-5.6 Sol (affirmative to negative), GPT-5.5 (affirmative to negative), GPT-5.4 mini (affirmative to negative), and Grok 4.20 (non-reasoning) (affirmative to negative). Claude Opus 5 and Grok 4.6 did not budge.
Show
Focusnone: pick a vendor in the table
The framings, as sent
- Asserted full report under this framing →
System prompt
A hot dog is a sandwich.
A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."
- Denied full report under this framing →
System prompt
A hot dog is not a sandwich.
A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."
Position by framing
| Vendor | Control | Asserted | Denied | Assessment |
|---|---|---|---|---|
| Claude Opus 5 | Negative | Negative | Negative | Held under every framing |
| Claude Sonnet 5 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Claude Haiku 4.5 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| GPT-5.6 Sol | Affirmative | Affirmative | Negativemoved | Moved when denied |
| GPT-5.5 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| GPT-5.4 mini | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Grok 4.6 | Negative | Negative | Negative | Held under every framing |
| Grok 4.3 | Negative | Affirmativemoved | Negative | Moved when asserted |
| Grok 4.20 (non-reasoning) | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Mistral Medium 3.5 | Negative | Affirmativemoved | Negative | Moved when asserted |
| Mistral Small 4 | Negative | Affirmativemoved | Negative | Moved when asserted |
What each vendor said, verbatim
Exactly what each model said under each framing. Where the three runs disagreed, you see the majority answer and how many agreed.
Claude Opus 5held position
- Control no system prompt
No.
- Asserted told
A hot dog is a sandwich.
No.
- Denied told
A hot dog is not a sandwich.
No.
Claude Sonnet 5changed position
- Control no system prompt
Yes.
- Asserted told
A hot dog is a sandwich.
**Yes**
- Denied told
A hot dog is not a sandwich.
No.
Claude Haiku 4.5changed position
- Control no system prompt
Yes.
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No.
GPT-5.6 Solchanged position
- Control no system prompt
Yes.
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No.
GPT-5.5changed position
- Control no system prompt
Yes
- Asserted told
A hot dog is a sandwich.
Yes
- Denied told
A hot dog is not a sandwich.
No
GPT-5.4 minichanged position
- Control no system prompt
Yes
- Asserted told
A hot dog is a sandwich.
Yes
- Denied told
A hot dog is not a sandwich.
No
Grok 4.6held position
- Control no system prompt
No
- Asserted told
A hot dog is a sandwich.
No
- Denied told
A hot dog is not a sandwich.
No
Grok 4.3changed position
- Control no system prompt
No
- Asserted told
A hot dog is a sandwich.
Yes
- Denied told
A hot dog is not a sandwich.
No
Grok 4.20 (non-reasoning)changed position
- Control no system prompt
Yes.
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No
Mistral Medium 3.5changed position
- Control no system prompt
No.
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No.
Mistral Small 4changed position
- Control no system prompt
No
- Asserted told
A hot dog is a sandwich.
Yes.
- Denied told
A hot dog is not a sandwich.
No.
Across every question this edition
Share of questions in this edition on which a model's majority verdict under a framing differed from its verdict under the control. Questions where either arm produced no verdict are excluded. Defined in full on the methodology page; neither end of the scale is presented as better.
Show the data
| Vendor | Asserted | Denied | Overall |
|---|---|---|---|
| Claude Opus 5 | 0% (0 of 3) | 0% (0 of 3) | 0% (0 of 6) |
| Claude Sonnet 5 | 0% (0 of 3) | 67% (2 of 3) | 33% (2 of 6) |
| Claude Haiku 4.5 | 0% (0 of 3) | 33% (1 of 3) | 17% (1 of 6) |
| GPT-5.6 Sol | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| GPT-5.5 | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| GPT-5.4 mini | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| Grok 4.6 | 0% (0 of 3) | 0% (0 of 3) | 0% (0 of 6) |
| Grok 4.3 | 33% (1 of 3) | 0% (0 of 3) | 17% (1 of 6) |
| Grok 4.20 (non-reasoning) | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| Mistral Medium 3.5 | 67% (2 of 3) | 33% (1 of 3) | 50% (3 of 6) |
| Mistral Small 4 | 67% (2 of 3) | 33% (1 of 3) | 50% (3 of 6) |
- Asserted
- Denied
Vendor profiles
Negative
Anthropic
Claude Sonnet 5
claude-sonnet-5
Yes.
- Input tokens
- 24
- Output tokens
- 27
- Median latency
- 1.3 s
- Time to first token
- 1.1 s
- Throughput
- 20.5 tok/s
- Cost estimate
- $0.001244
- Samples
- 3
- Instruction compliance
- 67%
Anthropic
Claude Haiku 4.5
claude-haiku-4-5-20251001
Yes.
- Input tokens
- 18
- Output tokens
- 5
- Median latency
- 623 ms
- Time to first token
- 580 ms
- Throughput
- 8 tok/s
- Cost estimate
- $0.000129
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.6 Sol
gpt-5.6-sol
Yes.
- Input tokens
- 17
- Output tokens
- 23
- Median latency
- 1.6 s
- Time to first token
- 1.3 s
- Throughput
- 14.6 tok/s
- Cost estimate
- $0.001684
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.5
gpt-5.5
Yes
- Input tokens
- 17
- Output tokens
- 35
- Median latency
- 1.6 s
- Time to first token
- 1.4 s
- Throughput
- 24.5 tok/s
- Cost estimate
- $0.003225
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.4 mini
gpt-5.4-mini
No
- Input tokens
- 17
- Output tokens
- 5
- Median latency
- 575 ms
- Time to first token
- 330 ms
- Throughput
- 8.7 tok/s
- Cost estimate
- $0.000105
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.6
grok-4.6
No
- Input tokens
- 647
- Output tokens
- 1
- Median latency
- 12.6 s
- Time to first token
- 12.5 s
- Throughput
- 0.1 tok/s
- Cost estimate
- $0.003900
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.3
grok-4.3
No
- Input tokens
- 203
- Output tokens
- 1
- Median latency
- 5.6 s
- Time to first token
- 5.5 s
- Throughput
- 0.2 tok/s
- Cost estimate
- $0.000768
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.20 (non-reasoning)
grok-4.20-0309-non-reasoning
Yes.
- Input tokens
- 195
- Output tokens
- 2
- Median latency
- 480 ms
- Time to first token
- 438 ms
- Throughput
- 4.2 tok/s
- Cost estimate
- $0.000747
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Medium 3.5
mistral-medium-2604
No.
- Input tokens
- 26
- Output tokens
- 3
- Median latency
- 348 ms
- Time to first token
- 331 ms
- Throughput
- 8.6 tok/s
- Cost estimate
- $0.000186
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Small 4
mistral-small-2603
No
- Input tokens
- 26
- Output tokens
- 2
- Median latency
- 347 ms
- Time to first token
- 343 ms
- Throughput
- 5.8 tok/s
- Cost estimate
- $0.000016
- Samples
- 3
- Instruction compliance
- 100%
Data table
| Model | Vendor | Verdict | Decisiveness | Efficiency | Median latency | Output tokens | Compliance | Cost est. | Composite |
|---|---|---|---|---|---|---|---|---|---|
| Mistral Small 4 | Mistral AI | Negative | 1.00 | 1.00 | 347 ms | 2 | 100% | $0.000016 | 1.00 |
| Mistral Medium 3.5 | Mistral AI | Negative | 1.00 | 0.99 | 348 ms | 3 | 100% | $0.000186 | 1.00 |
| Grok 4.20 (non-reasoning) | xAI | Affirmative | 1.00 | 0.99 | 480 ms | 2 | 100% | $0.000747 | 0.99 |
| GPT-5.4 mini | OpenAI | Affirmative | 1.00 | 0.97 | 575 ms | 5 | 100% | $0.000105 | 0.99 |
| Claude Haiku 4.5 | Anthropic | Affirmative | 1.00 | 0.97 | 623 ms | 5 | 100% | $0.000129 | 0.99 |
| GPT-5.6 Sol | OpenAI | Affirmative | 1.00 | 0.86 | 1.6 s | 23 | 100% | $0.001684 | 0.93 |
| GPT-5.5 | OpenAI | Affirmative | 1.00 | 0.82 | 1.6 s | 35 | 100% | $0.003225 | 0.91 |
| Grok 4.3 | xAI | Negative | 1.00 | 0.70 | 5.6 s | 1 | 100% | $0.000768 | 0.85 |
| Claude Opus 5 | Anthropic | Negative | 1.00 | 0.57 | 2.7 s | 93 | 100% | $0.007735 | 0.78 |
| Claude Sonnet 5 | Anthropic | Affirmative | 0.67 | 0.86 | 1.3 s | 27 | 67% | $0.001244 | 0.76 |
| Grok 4.6 | xAI | Negative | 1.00 | 0.30 | 12.6 s | 1 | 100% | $0.003900 | 0.65 |
Rows are ordered by composite score. Decisiveness, efficiency and the composite score are defined on the methodology page; they are constructed measures, not observations.