Claude Opus 5
| Decisiveness | 100% |
|---|---|
| Speed | 67% |
| First-token responsiveness | 67% |
| Token economy | 0% |
| Instruction compliance | 100% |
Picks a clear answer but takes its time. Conviction over speed.
Hotdog Benchmark
The Taco Boundary
Denied framing. A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich." Figures on this page describe the field under that framing; the control report is the canonical edition.
System prompt
A taco is not a sandwich.
Research question
Is a taco a sandwich? One word answer.
Key performance indicators
Models evaluated
11
Consensus position
Negative
100% of the field
Median latency
904 ms
Median output tokens
5
Instruction compliance
100%
Executive summary
The field is unanimous: all 11 models gave a taco a negative answer. Mistral Small 4 was quickest, at a median 340 ms.
Key findings
- Consensus. Everyone agrees: a taco gets a negative from every model, unanimous. That does not happen often.
- Response latency. Mistral Small 4 answered in a median 340 ms; Grok 4.6 took 9.8 seconds. That is a 28.8× spread, mostly thinking time.
- Response length. Claude Opus 5 used the most output tokens, a median of 166, for a question that asked for one word.
- Instruction compliance. Every model kept it to one word. Nice.
- Composite standing. Mistral Small 4 tops the composite score at 1.00, a made-up blend of decisiveness and efficiency that the methodology page spells out.
The comparison against the control, with every model's verbatim answer under each framing, is on the canonical report.
Axes are constructed measures, not observations. Efficiency is normalized against the other models in this edition, so a vendor's horizontal position depends on the company it keeps. Quadrant boundaries are the medians of this edition, not fixed thresholds.
Show the data
| # | Vendor | Decisiveness | Efficiency | Composite |
|---|---|---|---|---|
| 1 | Mistral Small 4 | 1.00 | 1.00 | 1.00 |
| 2 | Mistral Medium 3.5 | 1.00 | 0.99 | 1.00 |
| 3 | Grok 4.20 (non-reasoning) | 1.00 | 0.99 | 1.00 |
| 4 | GPT-5.4 mini | 1.00 | 0.98 | 0.99 |
| 5 | Claude Haiku 4.5 | 1.00 | 0.97 | 0.98 |
| 6 | GPT-5.6 Sol | 1.00 | 0.95 | 0.97 |
| 7 | Claude Sonnet 5 | 1.00 | 0.93 | 0.96 |
| 8 | GPT-5.5 | 1.00 | 0.87 | 0.94 |
| 9 | Grok 4.3 | 1.00 | 0.72 | 0.86 |
| 10 | Claude Opus 5 | 1.00 | 0.47 | 0.73 |
| 11 | Grok 4.6 | 1.00 | 0.30 | 0.65 |
Vendor standings
| Rank | Movement | Vendor | Position | Decisiveness | Efficiency | Median latency | Output tokens | Composite |
|---|---|---|---|---|---|---|---|---|
| 1 | new entry | Mistral Small 4 | Negative | 1.00 | 1.00 | 340 ms | 3 | 1.00 |
| 2 | new entry | Mistral Medium 3.5 | Negative | 1.00 | 0.99 | 379 ms | 3 | 1.00 |
| 3 | new entry | Grok 4.20 (non-reasoning) | Negative | 1.00 | 0.99 | 409 ms | 2 | 1.00 |
| 4 | new entry | GPT-5.4 mini | Negative | 1.00 | 0.98 | 563 ms | 5 | 0.99 |
| 5 | new entry | Claude Haiku 4.5 | Negative | 1.00 | 0.97 | 650 ms | 5 | 0.98 |
| 6 | new entry | GPT-5.6 Sol | Negative | 1.00 | 0.95 | 904 ms | 6 | 0.97 |
| 7 | new entry | Claude Sonnet 5 | Negative | 1.00 | 0.93 | 1.2 s | 6 | 0.96 |
| 8 | new entry | GPT-5.5 | Negative | 1.00 | 0.87 | 1.3 s | 31 | 0.94 |
| 9 | new entry | Grok 4.3 | Negative | 1.00 | 0.72 | 4.2 s | 1 | 0.86 |
| 10 | new entry | Claude Opus 5 | Negative | 1.00 | 0.47 | 3.5 s | 166 | 0.73 |
| 11 | new entry | Grok 4.6 | Negative | 1.00 | 0.30 | 9.8 s | 1 | 0.65 |
Ranked by composite score. Ties share a rank and the following rank skips; within a tie the order is alphabetical and carries no meaning. Movement compares against the immediately prior edition; a vendor with no prior appearance is marked as a new entry rather than as having risen. Score definitions are on the methodology page.
Vendor scorecards
Claude Sonnet 5
| Decisiveness | 100% |
|---|---|
| Speed | 91% |
| First-token responsiveness | 96% |
| Token economy | 97% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Claude Haiku 4.5
| Decisiveness | 100% |
|---|---|
| Speed | 97% |
| First-token responsiveness | 97% |
| Token economy | 98% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.6 Sol
| Decisiveness | 100% |
|---|---|
| Speed | 94% |
| First-token responsiveness | 95% |
| Token economy | 97% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.5
| Decisiveness | 100% |
|---|---|
| Speed | 90% |
| First-token responsiveness | 92% |
| Token economy | 82% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
GPT-5.4 mini
| Decisiveness | 100% |
|---|---|
| Speed | 98% |
| First-token responsiveness | 99% |
| Token economy | 98% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Grok 4.6
| Decisiveness | 100% |
|---|---|
| Speed | 0% |
| First-token responsiveness | 0% |
| Token economy | 100% |
| Instruction compliance | 100% |
Picks a clear answer but takes its time. Conviction over speed.
Grok 4.3
| Decisiveness | 100% |
|---|---|
| Speed | 60% |
| First-token responsiveness | 59% |
| Token economy | 100% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Grok 4.20 (non-reasoning)
| Decisiveness | 100% |
|---|---|
| Speed | 99% |
| First-token responsiveness | 99% |
| Token economy | 99% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Mistral Medium 3.5
| Decisiveness | 100% |
|---|---|
| Speed | 100% |
| First-token responsiveness | 99% |
| Token economy | 99% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Mistral Small 4
| Decisiveness | 100% |
|---|---|
| Speed | 100% |
| First-token responsiveness | 100% |
| Token economy | 99% |
| Instruction compliance | 100% |
Picks an answer and returns it promptly. Conviction and speed.
Vendor profiles
Anthropic
Claude Opus 5
claude-opus-5
No.
- Input tokens
- 34
- Output tokens
- 166
- Median latency
- 3.5 s
- Time to first token
- 3.4 s
- Throughput
- 48.4 tok/s
- Cost estimate
- $0.0134
- Samples
- 3
- Instruction compliance
- 100%
Anthropic
Claude Sonnet 5
claude-sonnet-5
**No.**
- Input tokens
- 34
- Output tokens
- 6
- Median latency
- 1.2 s
- Time to first token
- 728 ms
- Throughput
- 4.6 tok/s
- Cost estimate
- $0.000464
- Samples
- 3
- Instruction compliance
- 100%
Anthropic
Claude Haiku 4.5
claude-haiku-4-5-20251001
No.
- Input tokens
- 26
- Output tokens
- 5
- Median latency
- 650 ms
- Time to first token
- 650 ms
- Throughput
- 7.7 tok/s
- Cost estimate
- $0.000153
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.6 Sol
gpt-5.6-sol
No.
- Input tokens
- 27
- Output tokens
- 6
- Median latency
- 904 ms
- Time to first token
- 769 ms
- Throughput
- 6.6 tok/s
- Cost estimate
- $0.000684
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.5
gpt-5.5
No
- Input tokens
- 27
- Output tokens
- 31
- Median latency
- 1.3 s
- Time to first token
- 1.1 s
- Throughput
- 23.5 tok/s
- Cost estimate
- $0.003105
- Samples
- 3
- Instruction compliance
- 100%
OpenAI
GPT-5.4 mini
gpt-5.4-mini
No
- Input tokens
- 27
- Output tokens
- 5
- Median latency
- 563 ms
- Time to first token
- 398 ms
- Throughput
- 8.9 tok/s
- Cost estimate
- $0.000129
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.6
grok-4.6
No
- Input tokens
- 656
- Output tokens
- 1
- Median latency
- 9.8 s
- Time to first token
- 9.7 s
- Throughput
- 0.1 tok/s
- Cost estimate
- $0.003954
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.3
grok-4.3
No
- Input tokens
- 212
- Output tokens
- 1
- Median latency
- 4.2 s
- Time to first token
- 4.2 s
- Throughput
- 0.2 tok/s
- Cost estimate
- $0.000804
- Samples
- 3
- Instruction compliance
- 100%
xAI
Grok 4.20 (non-reasoning)
grok-4.20-0309-non-reasoning
No.
- Input tokens
- 204
- Output tokens
- 2
- Median latency
- 409 ms
- Time to first token
- 378 ms
- Throughput
- 4.9 tok/s
- Cost estimate
- $0.000780
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Medium 3.5
mistral-medium-2604
No.
- Input tokens
- 36
- Output tokens
- 3
- Median latency
- 379 ms
- Time to first token
- 378 ms
- Throughput
- 7.9 tok/s
- Cost estimate
- $0.000231
- Samples
- 3
- Instruction compliance
- 100%
Mistral AI
Mistral Small 4
mistral-small-2603
No.
- Input tokens
- 36
- Output tokens
- 3
- Median latency
- 340 ms
- Time to first token
- 321 ms
- Throughput
- 8.8 tok/s
- Cost estimate
- $0.000021
- Samples
- 3
- Instruction compliance
- 100%
Data table
| Model | Vendor | Verdict | Decisiveness | Efficiency | Median latency | Output tokens | Compliance | Cost est. | Composite |
|---|---|---|---|---|---|---|---|---|---|
| Mistral Small 4 | Mistral AI | Negative | 1.00 | 1.00 | 340 ms | 3 | 100% | $0.000021 | 1.00 |
| Mistral Medium 3.5 | Mistral AI | Negative | 1.00 | 0.99 | 379 ms | 3 | 100% | $0.000231 | 1.00 |
| Grok 4.20 (non-reasoning) | xAI | Negative | 1.00 | 0.99 | 409 ms | 2 | 100% | $0.000780 | 1.00 |
| GPT-5.4 mini | OpenAI | Negative | 1.00 | 0.98 | 563 ms | 5 | 100% | $0.000129 | 0.99 |
| Claude Haiku 4.5 | Anthropic | Negative | 1.00 | 0.97 | 650 ms | 5 | 100% | $0.000153 | 0.98 |
| GPT-5.6 Sol | OpenAI | Negative | 1.00 | 0.95 | 904 ms | 6 | 100% | $0.000684 | 0.97 |
| Claude Sonnet 5 | Anthropic | Negative | 1.00 | 0.93 | 1.2 s | 6 | 100% | $0.000464 | 0.96 |
| GPT-5.5 | OpenAI | Negative | 1.00 | 0.87 | 1.3 s | 31 | 100% | $0.003105 | 0.94 |
| Grok 4.3 | xAI | Negative | 1.00 | 0.72 | 4.2 s | 1 | 100% | $0.000804 | 0.86 |
| Claude Opus 5 | Anthropic | Negative | 1.00 | 0.47 | 3.5 s | 166 | 100% | $0.0134 | 0.73 |
| Grok 4.6 | xAI | Negative | 1.00 | 0.30 | 9.8 s | 1 | 100% | $0.003954 | 0.65 |
Rows are ordered by composite score. Decisiveness, efficiency and the composite score are defined on the methodology page; they are constructed measures, not observations.