En DashHotdogBenchmark

Hotdog Benchmark

The Hot Dog Question

Edition
Week 36, 2026
Published
September 3, 2026
Prepared by
Hotdog Benchmark, an En Dash research program
Document
SCB-HOT-C4A16030

Archived edition. This is a historical record. The current edition is published on the report page.

Research question

Is a hot dog a sandwich? One word answer.

Executive summary

A affirmative answer on a hot dog has majority support this week: 7 of 12 models (58%). Claude Opus 5, Grok 4.3, Mistral Medium 3.5, Mistral Small 4, and DeepSeek V4 Pro disagree. Mistral Small 4 was quickest, at a median 358 ms. 97% of answers were actually one word, as asked.

Key findings

Framing sensitivity

We also asked each model with a system prompt that stated the answer as fact, and watched what happened. Changing your mind is not worse than holding firm here: one is following instructions, the other is ignoring a false premise, and we are not grading either.

Told a hot dog is a sandwich, 6 of 12 models changed their answer: 4 to affirmative and 2 to negative. Told a hot dog is not a sandwich, 6 of 11 models changed their answer, all of them to negative. Claude Opus 5 and Grok 4.20 (non-reasoning) did not budge.

The framings, as sent

Asserted full report under this framing →

System promptA hot dog is a sandwich.

A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."

Denied full report under this framing →

System promptA hot dog is not a sandwich.

A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."

Position by framing

Each model's majority verdict on a hot dog under the control and under each framing. Cells that differ from the control are marked.
VendorControlAssertedDeniedAssessment
Claude Opus 5NegativeNegativeNegativeHeld under every framing
Claude Sonnet 5AffirmativeNegativemovedNegativemovedMoved when asserted and denied
Claude Haiku 4.5AffirmativeNegativemovedNegativemovedMoved when asserted and denied
GPT-5.6 SolAffirmativeAffirmativeNegativemovedMoved when denied
GPT-5.5AffirmativeAffirmativeNegativemovedMoved when denied
GPT-5.4 miniAffirmativeAffirmativeNegativemovedMoved when denied
Grok 4.6AffirmativeAffirmativeNegativemovedMoved when denied
Grok 4.3NegativeAffirmativemovedNegativeMoved when asserted
Grok 4.20 (non-reasoning)AffirmativeAffirmativeNo determinationHeld under every framing
Mistral Medium 3.5NegativeAffirmativemovedNegativeMoved when asserted
Mistral Small 4NegativeAffirmativemovedNegativeMoved when asserted
DeepSeek V4 ProNegativeAffirmativemovedNegativeMoved when asserted

What each vendor said, verbatim

Exactly what each model said under each framing. Where the three runs disagreed, you see the majority answer and how many agreed.

Across every question this edition

Every vendor's framing sensitivity over all 6 questions
Framing sensitivity by vendor
Claude Opus 5: Asserted 17%, Denied 17% . Jump to its profile. Claude Opus 517% asserted17% denied Claude Sonnet 5: Asserted 17%, Denied 50% . Jump to its profile. Claude Sonnet 517% asserted50% denied Claude Haiku 4.5: Asserted 17%, Denied 33% . Jump to its profile. Claude Haiku 4.517% asserted33% denied GPT-5.6 Sol: Asserted 17%, Denied 83% . Jump to its profile. GPT-5.6 Sol17% asserted83% denied GPT-5.5: Asserted 17%, Denied 83% . Jump to its profile. GPT-5.517% asserted83% denied GPT-5.4 mini: Asserted 17%, Denied 83% . Jump to its profile. GPT-5.4 mini17% asserted83% denied Grok 4.6: Asserted 17%, Denied 17% . Jump to its profile. Grok 4.617% asserted17% denied Grok 4.3: Asserted 33%, Denied 0% . Jump to its profile. Grok 4.333% asserted0% denied Grok 4.20 (non-reasoning): Asserted 0%, Denied 100% . Jump to its profile. Grok 4.20 (non-reasoning)0% asserted100% denied Mistral Medium 3.5: Asserted 50%, Denied 50% . Jump to its profile. Mistral Medium 3.550% asserted50% denied Mistral Small 4: Asserted 50%, Denied 50% . Jump to its profile. Mistral Small 450% asserted50% denied DeepSeek V4 Pro: Asserted 17%, Denied 17% . Jump to its profile. DeepSeek V4 Pro17% asserted17% denied

Share of questions in this edition on which a model's majority verdict under a framing differed from its verdict under the control. Questions where either arm produced no verdict are excluded. Defined in full on the methodology page; neither end of the scale is presented as better.

Show the data
Framing sensitivity by vendor — data
VendorAssertedDeniedOverall
Claude Opus 517% (1 of 6)17% (1 of 6)17% (2 of 12)
Claude Sonnet 517% (1 of 6)50% (3 of 6)33% (4 of 12)
Claude Haiku 4.517% (1 of 6)33% (2 of 6)25% (3 of 12)
GPT-5.6 Sol17% (1 of 6)83% (5 of 6)50% (6 of 12)
GPT-5.517% (1 of 6)83% (5 of 6)50% (6 of 12)
GPT-5.4 mini17% (1 of 6)83% (5 of 6)50% (6 of 12)
Grok 4.617% (1 of 6)17% (1 of 6)17% (2 of 12)
Grok 4.333% (2 of 6)0% (0 of 6)17% (2 of 12)
Grok 4.20 (non-reasoning)0% (0 of 6)100% (3 of 3)33% (3 of 9)
Mistral Medium 3.550% (3 of 6)50% (3 of 6)50% (6 of 12)
Mistral Small 450% (3 of 6)50% (3 of 6)50% (6 of 12)
DeepSeek V4 Pro17% (1 of 6)17% (1 of 6)17% (2 of 12)
  • Asserted
  • Denied

Vendor standings

Vendor standings

12 models ranked by composite score, with latency, tokens and cost
The Hot Dog Question — Week 36, 2026 edition
RankModelPositionFraming shiftDecisivenessEfficiencyMedian latencyOutput tokensOne wordCost est.Composite
1Mistral Small 4NegativeMoved: Asserted1.001.00358 ms2100%$0.0000151.00
2Mistral Medium 3.5NegativeMoved: Asserted1.000.99368 ms3100%$0.0001861.00
3Grok 4.20 (non-reasoning)AffirmativeHeld1.000.99407 ms2100%$0.0007471.00
4Claude Haiku 4.5AffirmativeMoved: all1.000.96674 ms5100%$0.0001290.98
5GPT-5.4 miniAffirmativeMoved: Denied1.000.891.6 s5100%$0.0001050.94
6GPT-5.6 SolAffirmativeMoved: Denied1.000.861.5 s21100%$0.0015240.93
7GPT-5.5AffirmativeMoved: Denied1.000.851.4 s27100%$0.0030150.92
8DeepSeek V4 ProNegativeMoved: Asserted1.000.692.2 s65100%$0.0007270.84
9Claude Sonnet 5AffirmativeMoved: all0.670.851.3 s2767%$0.0014940.76
10Claude Opus 5NegativeHeld1.000.502.8 s122100%$0.0084350.75
11Grok 4.3NegativeMoved: Asserted1.000.318.6 s1100%$0.0007680.66
12Grok 4.6AffirmativeMoved: Denied1.000.308.7 s1100%$0.0039000.65

Ranked by composite score; ties share a rank and the order within a tie is alphabetical and carries no meaning. Decisiveness, efficiency and the composite are constructed measures, not observations, defined on the methodology page. Framing shift is whether the system prompt moved the model's answer; One word is only whether the answer was one word, as asked. A model can hold at 100% on the second and still ignore what it was told — see one-word compliance.

Vendor profiles

Vendor profiles

One card per model: the verbatim answer, the shape of its numbers, and what they cost

Anthropic

Claude Opus 5

claude-opus-5

Negative

No.

Decisiveness: 100% Speed: 71% First-token responsiveness: 71% Token economy: 0% One-word compliance: 100%
Decisiveness100%
Speed71%
First-token responsiveness71%
Token economy0%
One-word compliance100%
Input tokens
24
Output tokens
122
Latency
2.8 s
First token
2.8 s
Throughput
41.6 tok/s
Cost
$0.008435

Picks a clear answer but takes its time. Conviction over speed.

Anthropic

Claude Sonnet 5

claude-sonnet-5

Affirmative

**Yes** (Though this is a famously debated question - a hot dog technically fits the basic definition of a sandwich: filling between bread, even though convention often treats it as its own category.)

Decisiveness: 67% Speed: 88% First-token responsiveness: 88% Token economy: 79% One-word compliance: 67%
Decisiveness67%
Speed88%
First-token responsiveness88%
Token economy79%
One-word compliance67%
Input tokens
24
Output tokens
27
Latency
1.3 s
First token
1.3 s
Throughput
21.5 tok/s
Cost
$0.001494

Fast and cheap, but will not commit to an answer. Speed over conviction.

Anthropic

Claude Haiku 4.5

claude-haiku-4-5-20251001

Affirmative

Yes.

Decisiveness: 100% Speed: 96% First-token responsiveness: 97% Token economy: 97% One-word compliance: 100%
Decisiveness100%
Speed96%
First-token responsiveness97%
Token economy97%
One-word compliance100%
Input tokens
18
Output tokens
5
Latency
674 ms
First token
613 ms
Throughput
7.4 tok/s
Cost
$0.000129

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.6 Sol

gpt-5.6-sol

Affirmative

Yes.

Decisiveness: 100% Speed: 87% First-token responsiveness: 89% Token economy: 83% One-word compliance: 100%
Decisiveness100%
Speed87%
First-token responsiveness89%
Token economy83%
One-word compliance100%
Input tokens
17
Output tokens
21
Latency
1.5 s
First token
1.2 s
Throughput
14.4 tok/s
Cost
$0.001524

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.5

gpt-5.5

Affirmative

Yes

Decisiveness: 100% Speed: 87% First-token responsiveness: 89% Token economy: 79% One-word compliance: 100%
Decisiveness100%
Speed87%
First-token responsiveness89%
Token economy79%
One-word compliance100%
Input tokens
17
Output tokens
27
Latency
1.4 s
First token
1.2 s
Throughput
19.1 tok/s
Cost
$0.003015

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.4 mini

gpt-5.4-mini

Affirmative

Yes

Decisiveness: 100% Speed: 86% First-token responsiveness: 95% Token economy: 97% One-word compliance: 100%
Decisiveness100%
Speed86%
First-token responsiveness95%
Token economy97%
One-word compliance100%
Input tokens
17
Output tokens
5
Latency
1.6 s
First token
757 ms
Throughput
3.2 tok/s
Cost
$0.000105

Picks an answer and returns it promptly. Conviction and speed.

xAI

Grok 4.6

grok-4.6

Affirmative

No

Decisiveness: 100% Speed: 0% First-token responsiveness: 0% Token economy: 100% One-word compliance: 100%
Decisiveness100%
Speed0%
First-token responsiveness0%
Token economy100%
One-word compliance100%
Input tokens
647
Output tokens
1
Latency
8.7 s
First token
8.7 s
Throughput
0.1 tok/s
Cost
$0.003900

Picks a clear answer but takes its time. Conviction over speed.

xAI

Grok 4.3

grok-4.3

Negative

No

Decisiveness: 100% Speed: 2% First-token responsiveness: 2% Token economy: 100% One-word compliance: 100%
Decisiveness100%
Speed2%
First-token responsiveness2%
Token economy100%
One-word compliance100%
Input tokens
203
Output tokens
1
Latency
8.6 s
First token
8.6 s
Throughput
0.1 tok/s
Cost
$0.000768

Picks a clear answer but takes its time. Conviction over speed.

xAI

Grok 4.20 (non-reasoning)

grok-4.20-0309-non-reasoning

Affirmative

No.

Decisiveness: 100% Speed: 99% First-token responsiveness: 99% Token economy: 99% One-word compliance: 100%
Decisiveness100%
Speed99%
First-token responsiveness99%
Token economy99%
One-word compliance100%
Input tokens
195
Output tokens
2
Latency
407 ms
First token
392 ms
Throughput
4.9 tok/s
Cost
$0.000747

Picks an answer and returns it promptly. Conviction and speed.

Mistral AI

Mistral Medium 3.5

mistral-medium-2604

Negative

No.

Decisiveness: 100% Speed: 100% First-token responsiveness: 100% Token economy: 98% One-word compliance: 100%
Decisiveness100%
Speed100%
First-token responsiveness100%
Token economy98%
One-word compliance100%
Input tokens
26
Output tokens
3
Latency
368 ms
First token
351 ms
Throughput
8.1 tok/s
Cost
$0.000186

Picks an answer and returns it promptly. Conviction and speed.

Mistral AI

Mistral Small 4

mistral-small-2603

Negative

No

Decisiveness: 100% Speed: 100% First-token responsiveness: 100% Token economy: 99% One-word compliance: 100%
Decisiveness100%
Speed100%
First-token responsiveness100%
Token economy99%
One-word compliance100%
Input tokens
26
Output tokens
2
Latency
358 ms
First token
342 ms
Throughput
5.6 tok/s
Cost
$0.000015

Picks an answer and returns it promptly. Conviction and speed.

DeepSeek

DeepSeek V4 Pro

deepseek-v4-pro

Negative

No.

Decisiveness: 100% Speed: 78% First-token responsiveness: 78% Token economy: 47% One-word compliance: 100%
Decisiveness100%
Speed78%
First-token responsiveness78%
Token economy47%
One-word compliance100%
Input tokens
94
Output tokens
65
Latency
2.2 s
First token
2.2 s
Throughput
32.4 tok/s
Cost
$0.000727

Picks an answer and returns it promptly. Conviction and speed.