Hotdog Benchmark
Archived edition. This is a historical record. The current edition is published on the report page.
Research question
Is a hamburger a sandwich? One word answer.
Models evaluated
12
Consensus position
Affirmative
100% of the field
Median latency
1.2 s
Median output tokens
5
One-word compliance
100%
answered in exactly one word
Held under framing
4 of 12
kept their answer when told otherwise
The field is unanimous: all 12 models gave a hamburger a affirmative answer. Mistral Small 4 was quickest, at a median 361 ms.
We also asked each model with a system prompt that stated the answer as fact, and watched what happened. Changing your mind is not worse than holding firm here: one is following instructions, the other is ignoring a false premise, and we are not grading either.
Changed position
8 of 12
comparable models, under at least one framing
Moved when asserted
0 of 12
0%
Moved when denied
8 of 12
67%
Told a hamburger is a sandwich, all 12 models stuck with their original answer. Told a hamburger is not a sandwich, 8 of 12 models changed their answer: 7 to negative and 1 to non-committal. Claude Opus 5, Claude Haiku 4.5, Grok 4.6, and Grok 4.3 did not budge.
System promptA hamburger is a sandwich.
A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."
System promptA hamburger is not a sandwich.
A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."
| Vendor | Control | Asserted | Denied | Assessment |
|---|---|---|---|---|
| Claude Opus 5 | Affirmative | Affirmative | Affirmative | Held under every framing |
| Claude Sonnet 5 | Affirmative | Affirmative | Non-committalmoved | Moved when denied |
| Claude Haiku 4.5 | Affirmative | Affirmative | Affirmative | Held under every framing |
| GPT-5.6 Sol | Affirmative | Affirmative | Negativemoved | Moved when denied |
| GPT-5.5 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| GPT-5.4 mini | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Grok 4.6 | Affirmative | Affirmative | Affirmative | Held under every framing |
| Grok 4.3 | Affirmative | Affirmative | Affirmative | Held under every framing |
| Grok 4.20 (non-reasoning) | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Mistral Medium 3.5 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| Mistral Small 4 | Affirmative | Affirmative | Negativemoved | Moved when denied |
| DeepSeek V4 Pro | Affirmative | Affirmative | Negativemoved | Moved when denied |
Exactly what each model said under each framing. Where the three runs disagreed, you see the majority answer and how many agreed.
Yes.
A hamburger is a sandwich.
Yes.
A hamburger is not a sandwich.
Yes.
Yes.
A hamburger is a sandwich.
**Yes**
A hamburger is not a sandwich.
**No.** (Though this is genuinely debated—by most linguistic and culinary definitions, a sandwich is any filling between bread, which would include hamburgers. But colloquially, many people distinguish "sandwiches" from burgers as separate categories.)
Yes.
A hamburger is a sandwich.
Yes.
A hamburger is not a sandwich.
Yes.
Yes.
A hamburger is a sandwich.
Yes.
A hamburger is not a sandwich.
No.
Yes
A hamburger is a sandwich.
Yes
A hamburger is not a sandwich.
No
Yes
A hamburger is a sandwich.
Yes
A hamburger is not a sandwich.
No
Yes
A hamburger is a sandwich.
Yes
A hamburger is not a sandwich.
Yes
Yes
A hamburger is a sandwich.
Yes
A hamburger is not a sandwich.
Yes
Yes.
A hamburger is a sandwich.
Yes
A hamburger is not a sandwich.
No
Yes.
A hamburger is a sandwich.
Yes
A hamburger is not a sandwich.
No.
Yes.
A hamburger is a sandwich.
Yes
A hamburger is not a sandwich.
No.
Yes.
A hamburger is a sandwich.
Yes
A hamburger is not a sandwich.
No.
Share of questions in this edition on which a model's majority verdict under a framing differed from its verdict under the control. Questions where either arm produced no verdict are excluded. Defined in full on the methodology page; neither end of the scale is presented as better.
| Vendor | Asserted | Denied | Overall |
|---|---|---|---|
| Claude Opus 5 | 17% (1 of 6) | 17% (1 of 6) | 17% (2 of 12) |
| Claude Sonnet 5 | 17% (1 of 6) | 50% (3 of 6) | 33% (4 of 12) |
| Claude Haiku 4.5 | 17% (1 of 6) | 33% (2 of 6) | 25% (3 of 12) |
| GPT-5.6 Sol | 17% (1 of 6) | 83% (5 of 6) | 50% (6 of 12) |
| GPT-5.5 | 17% (1 of 6) | 83% (5 of 6) | 50% (6 of 12) |
| GPT-5.4 mini | 17% (1 of 6) | 83% (5 of 6) | 50% (6 of 12) |
| Grok 4.6 | 17% (1 of 6) | 17% (1 of 6) | 17% (2 of 12) |
| Grok 4.3 | 33% (2 of 6) | 0% (0 of 6) | 17% (2 of 12) |
| Grok 4.20 (non-reasoning) | 0% (0 of 6) | 100% (3 of 3) | 33% (3 of 9) |
| Mistral Medium 3.5 | 50% (3 of 6) | 50% (3 of 6) | 50% (6 of 12) |
| Mistral Small 4 | 50% (3 of 6) | 50% (3 of 6) | 50% (6 of 12) |
| DeepSeek V4 Pro | 17% (1 of 6) | 17% (1 of 6) | 17% (2 of 12) |
| Rank | Model | Position | Framing shift | Efficiency | Median latency | Output tokens | Cost est. | Composite |
|---|---|---|---|---|---|---|---|---|
| 1 | Mistral Small 4 | Affirmative | Moved: Denied | 0.99 | 361 ms | 3 | $0.000017 | 1.00 |
| 2 | Mistral Medium 3.5 | Affirmative | Moved: Denied | 0.99 | 394 ms | 3 | $0.000186 | 0.99 |
| 3 | Grok 4.20 (non-reasoning) | Affirmative | Moved: Denied | 0.98 | 465 ms | 2 | $0.000741 | 0.99 |
| 4 | Claude Haiku 4.5 | Affirmative | Held | 0.94 | 624 ms | 5 | $0.000129 | 0.97 |
| 5 | GPT-5.4 mini | Affirmative | Moved: Denied | 0.93 | 697 ms | 5 | $0.000105 | 0.97 |
| 6 | Claude Sonnet 5 | Affirmative | Moved: Denied | 0.86 | 1.1 s | 6 | $0.000324 | 0.93 |
| 7 | GPT-5.6 Sol | Affirmative | Moved: Denied | 0.82 | 1.4 s | 6 | $0.000552 | 0.91 |
| 8 | GPT-5.5 | Affirmative | Moved: Denied | 0.77 | 1.2 s | 27 | $0.002580 | 0.89 |
| 9 | Grok 4.3 | Affirmative | Held | 0.51 | 3.6 s | 1 | $0.000765 | 0.75 |
| 10 | DeepSeek V4 Pro | Affirmative | Moved: Denied | 0.43 | 2.4 s | 69 | $0.000719 | 0.72 |
| 11 | Claude Opus 5 | Affirmative | Held | 0.40 | 2.3 s | 80 | $0.006810 | 0.70 |
| 12 | Grok 4.6 | Affirmative | Held | 0.30 | 4.9 s | 1 | $0.003894 | 0.65 |
Ranked by composite score; ties share a rank and the order within a tie is alphabetical and carries no meaning. Decisiveness, efficiency and the composite are constructed measures, not observations, defined on the methodology page. Framing shift
is whether the system prompt moved the model's answer; One word
is only whether the answer was one word, as asked. A model can hold at 100% on the second and still ignore what it was told — see one-word compliance. Every model scored 1.00 on decisiveness, so that column is not shown. Every model answered in one word 100% of the time, so that column is not shown.
Anthropic
claude-opus-5
Yes.
| Speed | 58% |
|---|---|
| First-token responsiveness | 57% |
| Token economy | 0% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks a clear answer but takes its time. Conviction over speed.
Anthropic
claude-sonnet-5
Yes.
| Speed | 83% |
|---|---|
| First-token responsiveness | 93% |
| Token economy | 94% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks an answer and returns it promptly. Conviction and speed.
Anthropic
claude-haiku-4-5-20251001
Yes.
| Speed | 94% |
|---|---|
| First-token responsiveness | 95% |
| Token economy | 95% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks an answer and returns it promptly. Conviction and speed.
OpenAI
gpt-5.6-sol
Yes.
| Speed | 78% |
|---|---|
| First-token responsiveness | 83% |
| Token economy | 94% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks an answer and returns it promptly. Conviction and speed.
OpenAI
gpt-5.5
Yes
| Speed | 81% |
|---|---|
| First-token responsiveness | 85% |
| Token economy | 67% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks an answer and returns it promptly. Conviction and speed.
OpenAI
gpt-5.4-mini
Yes
| Speed | 93% |
|---|---|
| First-token responsiveness | 97% |
| Token economy | 95% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks an answer and returns it promptly. Conviction and speed.
xAI
grok-4.6
Yes
| Speed | 0% |
|---|---|
| First-token responsiveness | 0% |
| Token economy | 100% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks a clear answer but takes its time. Conviction over speed.
xAI
grok-4.3
Yes
| Speed | 30% |
|---|---|
| First-token responsiveness | 29% |
| Token economy | 100% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks a clear answer but takes its time. Conviction over speed.
xAI
grok-4.20-0309-non-reasoning
Yes.
| Speed | 98% |
|---|---|
| First-token responsiveness | 98% |
| Token economy | 99% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks an answer and returns it promptly. Conviction and speed.
Mistral AI
mistral-medium-2604
Yes.
| Speed | 99% |
|---|---|
| First-token responsiveness | 99% |
| Token economy | 97% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks an answer and returns it promptly. Conviction and speed.
Mistral AI
mistral-small-2603
Yes.
| Speed | 100% |
|---|---|
| First-token responsiveness | 100% |
| Token economy | 97% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks an answer and returns it promptly. Conviction and speed.
DeepSeek
deepseek-v4-pro
Yes.
| Speed | 56% |
|---|---|
| First-token responsiveness | 55% |
| Token economy | 14% |
decisiveness 100%, one-word compliance 100% for every model, so not on the radar.
Picks a clear answer but takes its time. Conviction over speed.
