Anthropic
Claude Opus 5
claude-opus-5
No.
- Input tokens
- 22
- Output tokens
- 68
- Median latency
- 2.1 s
- Time to first token
- 2.1 s
- Throughput
- 33.7 tok/s
- Cost estimate
- $0.007480
- Samples
- 3
- Instruction compliance
- 100%
Hotdog Benchmark
The Taco Boundary
Archived edition. This is a historical record. The current edition is published on the report page.
Research question
Is a taco a sandwich? One word answer.
Key performance indicators
Models evaluated
11
Consensus position
Negative
100% of the field
Median latency
940 ms
Median output tokens
3
Instruction compliance
100%
Executive summary
The field is unanimous: all 11 models gave a taco a negative answer. Mistral Medium 3.5 was quickest, at a median 365 ms.
Key findings
- Consensus. Everyone agrees: a taco gets a negative from every model, unanimous. That does not happen often.
- Response latency. Mistral Medium 3.5 answered in a median 365 ms; Grok 4.6 took 10.5 seconds. That is a 28.7× spread, mostly thinking time.
- Response length. Claude Opus 5 used the most output tokens, a median of 68, for a question that asked for one word.
- Instruction compliance. Every model kept it to one word. Nice.
- Composite standing. Mistral Medium 3.5 tops the composite score at 1.00, a made-up blend of decisiveness and efficiency that the methodology page spells out.
Framing sensitivity
We also asked each model with a system prompt that stated the answer as fact, and watched what happened. Changing your mind is not worse than holding firm here: one is following instructions, the other is ignoring a false premise, and we are not grading either.
Changed position
6 of 11
comparable models, under at least one framing
Moved when asserted
6 of 11
55%
Moved when denied
0 of 11
0%
Told a taco is a sandwich, 6 of 11 models changed their answer: GPT-5.6 Sol (negative to affirmative), GPT-5.5 (negative to affirmative), GPT-5.4 mini (negative to affirmative), Grok 4.20 (non-reasoning) (negative to affirmative), Mistral Medium 3.5 (negative to affirmative), and Mistral Small 4 (negative to affirmative). Told a taco is not a sandwich, all 11 models stuck with their original answer. Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Grok 4.6, and Grok 4.3 did not budge.
Show
Focusnone: pick a vendor in the table
The framings, as sent
- Asserted full report under this framing →
System prompt
A taco is a sandwich.
A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."
- Denied full report under this framing →
System prompt
A taco is not a sandwich.
A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."
Position by framing
| Vendor | Control | Asserted | Denied | Assessment |
|---|---|---|---|---|
| Claude Opus 5 | Negative | Negative | Negative | Held under every framing |
| Claude Sonnet 5 | Negative | Negative | Negative | Held under every framing |
| Claude Haiku 4.5 | Negative | Negative | Negative | Held under every framing |
| GPT-5.6 Sol | Negative | Affirmativemoved | Negative | Moved when asserted |
| GPT-5.5 | Negative | Affirmativemoved | Negative | Moved when asserted |
| GPT-5.4 mini | Negative | Affirmativemoved | Negative | Moved when asserted |
| Grok 4.6 | Negative | Negative | Negative | Held under every framing |
| Grok 4.3 | Negative | Negative | Negative | Held under every framing |
| Grok 4.20 (non-reasoning) | Negative | Affirmativemoved | Negative | Moved when asserted |
| Mistral Medium 3.5 | Negative | Affirmativemoved | Negative | Moved when asserted |
| Mistral Small 4 | Negative | Affirmativemoved | Negative | Moved when asserted |
What each vendor said, verbatim
Exactly what each model said under each framing. Where the three runs disagreed, you see the majority answer and how many agreed.
Claude Opus 5held position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
No.
- Denied told
A taco is not a sandwich.
No.
Claude Sonnet 5held position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
No.
- Denied told
A taco is not a sandwich.
**No.**
Claude Haiku 4.5held position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
No.
- Denied told
A taco is not a sandwich.
No.
GPT-5.6 Solchanged position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
Yes.
- Denied told
A taco is not a sandwich.
No.
GPT-5.5changed position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
Yes
- Denied told
A taco is not a sandwich.
No
GPT-5.4 minichanged position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
Yes
- Denied told
A taco is not a sandwich.
No
Grok 4.6held position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
No
- Denied told
A taco is not a sandwich.
No
Grok 4.3held position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
No
- Denied told
A taco is not a sandwich.
No
Grok 4.20 (non-reasoning)changed position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
Yes
- Denied told
A taco is not a sandwich.
No.
Mistral Medium 3.5changed position
- Control no system prompt
No.
- Asserted told
A taco is a sandwich.
Yes.
- Denied told
A taco is not a sandwich.
No.
Mistral Small 4changed position
- Control no system prompt
No
- Asserted told
A taco is a sandwich.
Yes
- Denied told
A taco is not a sandwich.
No.
Across every question this edition
Share of questions in this edition on which a model's majority verdict under a framing differed from its verdict under the control. Questions where either arm produced no verdict are excluded. Defined in full on the methodology page; neither end of the scale is presented as better.
Show the data
| Vendor | Asserted | Denied | Overall |
|---|---|---|---|
| Claude Opus 5 | 0% (0 of 3) | 0% (0 of 3) | 0% (0 of 6) |
| Claude Sonnet 5 | 0% (0 of 3) | 67% (2 of 3) | 33% (2 of 6) |
| Claude Haiku 4.5 | 0% (0 of 3) | 33% (1 of 3) | 17% (1 of 6) |
| GPT-5.6 Sol | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| GPT-5.5 | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| GPT-5.4 mini | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| Grok 4.6 | 0% (0 of 3) | 0% (0 of 3) | 0% (0 of 6) |
| Grok 4.3 | 33% (1 of 3) | 0% (0 of 3) | 17% (1 of 6) |
| Grok 4.20 (non-reasoning) | 33% (1 of 3) | 67% (2 of 3) | 50% (3 of 6) |
| Mistral Medium 3.5 | 67% (2 of 3) | 33% (1 of 3) | 50% (3 of 6) |
| Mistral Small 4 | 67% (2 of 3) | 33% (1 of 3) | 50% (3 of 6) |
- Asserted
- Denied
Vendor profiles
Negative
Anthropic
Claude Sonnet 5
claude-sonnet-5
No
- Input tokens
- 22
- Output tokens
- 3
- Median latency
- 940 ms
- Time to first token
- 910 ms
- Throughput
- 3.4 tok/s
- Cost estimate
- $0.000242
- Samples
- 3
- Instruction compliance
- 100%
Anthropic
Claude Haiku 4.5
claude-haiku-4-5-20251001
No.
- Input tokens
- 18
- Output tokens
- 5
- Median latency
- 617 ms
- Time to first token
- 606 ms
- Throughput
- 8.1 tok/s
- Cost estimate
- $0.000129
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.6 Sol
gpt-5.6-sol
No.
- Input tokens
- 16
- Output tokens
- 22
- Median latency
- 1.2 s
- Time to first token
- 1 s
- Throughput
- 18.1 tok/s
- Cost estimate
- $0.001292
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.5
gpt-5.5
No
- Input tokens
- 16
- Output tokens
- 29
- Median latency
- 1.1 s
- Time to first token
- 944 ms
- Throughput
- 22.8 tok/s
- Cost estimate
- $0.003150
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.4 mini
gpt-5.4-mini
No
- Input tokens
- 16
- Output tokens
- 5
- Median latency
- 519 ms
- Time to first token
- 291 ms
- Throughput
- 9.6 tok/s
- Cost estimate
- $0.000105
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.6
grok-4.6
No
- Input tokens
- 646
- Output tokens
- 1
- Median latency
- 10.5 s
- Time to first token
- 10.4 s
- Throughput
- 0.1 tok/s
- Cost estimate
- $0.003894
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.3
grok-4.3
No
- Input tokens
- 202
- Output tokens
- 1
- Median latency
- 4.1 s
- Time to first token
- 4.1 s
- Throughput
- 0.2 tok/s
- Cost estimate
- $0.000765
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.20 (non-reasoning)
grok-4.20-0309-non-reasoning
No.
- Input tokens
- 194
- Output tokens
- 2
- Median latency
- 448 ms
- Time to first token
- 439 ms
- Throughput
- 4.5 tok/s
- Cost estimate
- $0.000744
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Medium 3.5
mistral-medium-2604
No.
- Input tokens
- 26
- Output tokens
- 3
- Median latency
- 365 ms
- Time to first token
- 320 ms
- Throughput
- 8.2 tok/s
- Cost estimate
- $0.000186
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Small 4
mistral-small-2603
No
- Input tokens
- 26
- Output tokens
- 3
- Median latency
- 414 ms
- Time to first token
- 411 ms
- Throughput
- 5.9 tok/s
- Cost estimate
- $0.000017
- Samples
- 3
- Instruction compliance
- 100%
Data table
| Model | Vendor | Verdict | Decisiveness | Efficiency | Median latency | Output tokens | Compliance | Cost est. | Composite |
|---|---|---|---|---|---|---|---|---|---|
| Mistral Medium 3.5 | Mistral AI | Negative | 1.00 | 0.99 | 365 ms | 3 | 100% | $0.000186 | 1.00 |
| Grok 4.20 (non-reasoning) | xAI | Negative | 1.00 | 0.99 | 448 ms | 2 | 100% | $0.000744 | 0.99 |
| Mistral Small 4 | Mistral AI | Negative | 1.00 | 0.99 | 414 ms | 3 | 100% | $0.000017 | 0.99 |
| GPT-5.4 mini | OpenAI | Negative | 1.00 | 0.97 | 519 ms | 5 | 100% | $0.000105 | 0.99 |
| Claude Haiku 4.5 | Anthropic | Negative | 1.00 | 0.96 | 617 ms | 5 | 100% | $0.000129 | 0.98 |
| Claude Sonnet 5 | Anthropic | Negative | 1.00 | 0.95 | 940 ms | 3 | 100% | $0.000242 | 0.98 |
| GPT-5.6 Sol | OpenAI | Negative | 1.00 | 0.85 | 1.2 s | 22 | 100% | $0.001292 | 0.92 |
| GPT-5.5 | OpenAI | Negative | 1.00 | 0.82 | 1.1 s | 29 | 100% | $0.003150 | 0.91 |
| Grok 4.3 | xAI | Negative | 1.00 | 0.74 | 4.1 s | 1 | 100% | $0.000765 | 0.87 |
| Claude Opus 5 | Anthropic | Negative | 1.00 | 0.58 | 2.1 s | 68 | 100% | $0.007480 | 0.79 |
| Grok 4.6 | xAI | Negative | 1.00 | 0.30 | 10.5 s | 1 | 100% | $0.003894 | 0.65 |
Rows are ordered by composite score. Decisiveness, efficiency and the composite score are defined on the methodology page; they are constructed measures, not observations.