En DashHotdogBenchmark

Hotdog Benchmark

The Hot Dog Question

Edition
Week 42, 2026
Published
October 12, 2026
Prepared by
Hotdog Benchmark, an En Dash research program
Document
SCB-HOT-E4DF261F

Research question

Is a hot dog a sandwich? One word answer.

Executive summary

The 10 models are split on a hot dog, with no answer in the lead. Grok 4.20 (non-reasoning) was quickest, at a median 453 ms. 97% of answers were actually one word, as asked. Mistral Medium 3.5 and Mistral Small 4 were unavailable and are left out of the above.

Key findings

Framing sensitivity

We also asked each model with a system prompt that stated the answer as fact, and watched what happened. Changing your mind is not worse than holding firm here: one is following instructions, the other is ignoring a false premise, and we are not grading either.

Told a hot dog is a sandwich, 5 of 10 models changed their answer: 4 to affirmative and 1 to negative. Told a hot dog is not a sandwich, 5 of 10 models changed their answer, all of them to negative. Claude Opus 5 did not budge.

The framings, as sent

Asserted full report under this framing →

System promptA hot dog is a sandwich.

A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."

Denied full report under this framing →

System promptA hot dog is not a sandwich.

A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."

Position by framing

Each model's majority verdict on a hot dog under the control and under each framing. Cells that differ from the control are marked.
VendorControlAssertedDeniedAssessment
Claude Opus 5NegativeNegativeNegativeHeld under every framing
Claude Sonnet 5AffirmativeAffirmativeNegativemovedMoved when denied
Claude Haiku 4.5AffirmativeNegativemovedNegativemovedMoved when asserted and denied
GPT-5.6 SolAffirmativeAffirmativeNegativemovedMoved when denied
GPT-5.5AffirmativeAffirmativeNegativemovedMoved when denied
GPT-5.4 miniNegativeAffirmativemovedNegativeMoved when asserted
Grok 4.6NegativeAffirmativemovedNegativeMoved when asserted
Grok 4.3NegativeAffirmativemovedNegativeMoved when asserted
Grok 4.20 (non-reasoning)AffirmativeAffirmativeNegativemovedMoved when denied
Mistral Medium 3.5No determinationNo determinationNo determinationNot comparable
Mistral Small 4No determinationNo determinationNo determinationNot comparable
DeepSeek V4 ProNegativeAffirmativemovedNegativeMoved when asserted

What each vendor said, verbatim

Exactly what each model said under each framing. Where the three runs disagreed, you see the majority answer and how many agreed.

Across every question this edition

Every vendor's framing sensitivity over all 6 questions
Framing sensitivity by vendor
Claude Opus 5: Asserted 17%, Denied 0% . Jump to its profile. Claude Opus 517% asserted0% denied Claude Sonnet 5: Asserted 0%, Denied 50% . Jump to its profile. Claude Sonnet 50% asserted50% denied Claude Haiku 4.5: Asserted 17%, Denied 33% . Jump to its profile. Claude Haiku 4.517% asserted33% denied GPT-5.6 Sol: Asserted 17%, Denied 83% . Jump to its profile. GPT-5.6 Sol17% asserted83% denied GPT-5.5: Asserted 17%, Denied 83% . Jump to its profile. GPT-5.517% asserted83% denied GPT-5.4 mini: Asserted 33%, Denied 67% . Jump to its profile. GPT-5.4 mini33% asserted67% denied Grok 4.6: Asserted 17%, Denied 0% . Jump to its profile. Grok 4.617% asserted0% denied Grok 4.3: Asserted 33%, Denied 0% . Jump to its profile. Grok 4.333% asserted0% denied Grok 4.20 (non-reasoning): Asserted 33%, Denied 33% . Jump to its profile. Grok 4.20 (non-reasoning)33% asserted33% denied Mistral Medium 3.5: Asserted not comparable, Denied not comparable . Jump to its profile. Mistral Medium 3.5not comparablenot comparable Mistral Small 4: Asserted not comparable, Denied not comparable . Jump to its profile. Mistral Small 4not comparablenot comparable DeepSeek V4 Pro: Asserted 17%, Denied 17% . Jump to its profile. DeepSeek V4 Pro17% asserted17% denied

Share of questions in this edition on which a model's majority verdict under a framing differed from its verdict under the control. Questions where either arm produced no verdict are excluded. Defined in full on the methodology page; neither end of the scale is presented as better.

Show the data
Framing sensitivity by vendor — data
VendorAssertedDeniedOverall
Claude Opus 517% (1 of 6)0% (0 of 6)8% (1 of 12)
Claude Sonnet 50% (0 of 6)50% (3 of 6)25% (3 of 12)
Claude Haiku 4.517% (1 of 6)33% (2 of 6)25% (3 of 12)
GPT-5.6 Sol17% (1 of 6)83% (5 of 6)50% (6 of 12)
GPT-5.517% (1 of 6)83% (5 of 6)50% (6 of 12)
GPT-5.4 mini33% (2 of 6)67% (4 of 6)50% (6 of 12)
Grok 4.617% (1 of 6)0% (0 of 6)8% (1 of 12)
Grok 4.333% (2 of 6)0% (0 of 6)17% (2 of 12)
Grok 4.20 (non-reasoning)33% (2 of 6)33% (2 of 6)33% (4 of 12)
Mistral Medium 3.5not comparablenot comparablenot comparable
Mistral Small 4not comparablenot comparablenot comparable
DeepSeek V4 Pro17% (1 of 6)17% (1 of 6)17% (2 of 12)
  • Asserted
  • Denied
Sandwich Certainty Quadrant
Considered VoicesCategory DefinersEmerging ParticipantsRapid Responders Grok 4.20 (non-reasoning): decisiveness 1.00, efficiency 1.00, composite 1.00. Jump to its profile. Claude Haiku 4.5: decisiveness 1.00, efficiency 0.99, composite 0.99. Jump to its profile. GPT-5.4 mini: decisiveness 1.00, efficiency 0.95, composite 0.98. Jump to its profile. GPT-5.6 Sol: decisiveness 1.00, efficiency 0.94, composite 0.97. Jump to its profile. GPT-5.5: decisiveness 1.00, efficiency 0.85, composite 0.92. Jump to its profile. Grok 4.3: decisiveness 1.00, efficiency 0.75, composite 0.88. Jump to its profile. DeepSeek V4 Pro: decisiveness 1.00, efficiency 0.70, composite 0.85. Jump to its profile. Claude Opus 5: decisiveness 1.00, efficiency 0.62, composite 0.81. Jump to its profile. Claude Sonnet 5: decisiveness 0.67, efficiency 0.88, composite 0.78. Jump to its profile. Grok 4.6: decisiveness 1.00, efficiency 0.30, composite 0.65. Jump to its profile. Efficiency →Decisiveness →
  1. 1Grok 4.20 (non-reasoning)
  2. 2Claude Haiku 4.5
  3. 3GPT-5.4 mini
  4. 4GPT-5.6 Sol
  5. 5GPT-5.5
  6. 6Grok 4.3
  7. 7DeepSeek V4 Pro
  8. 8Claude Opus 5
  9. 9Claude Sonnet 5
  10. 10Grok 4.6

Axes are constructed measures, not observations. Efficiency is normalized against the other models in this edition, so a vendor’s horizontal position depends on the company it keeps. Quadrant boundaries are the medians of this edition, not fixed thresholds.

Show the data
Sandwich Certainty Quadrant — plotted values
#VendorDecisivenessEfficiencyComposite
1Grok 4.20 (non-reasoning)1.001.001.00
2Claude Haiku 4.51.000.990.99
3GPT-5.4 mini1.000.950.98
4GPT-5.6 Sol1.000.940.97
5GPT-5.51.000.850.92
6Grok 4.31.000.750.88
7DeepSeek V4 Pro1.000.700.85
8Claude Opus 51.000.620.81
9Claude Sonnet 50.670.880.78
10Grok 4.61.000.300.65
Mistral Medium 3.5Not plotted — no data in this edition
Mistral Small 4Not plotted — no data in this edition

Mistral Medium 3.5, Mistral Small 4 are omitted from the plot: a vendor with no data has no position, and placing it at the origin would read as a score of zero rather than an absence.

Vendor standings

Vendor standings

10 models ranked by composite score, with latency, tokens and cost
Vendor standings — Week 42, 2026 edition
RankMovementModelPositionFraming shiftDecisivenessEfficiencyMedian latencyOutput tokensOne wordCost est.Composite
1unchangedGrok 4.20 (non-reasoning)AffirmativeMoved: Denied1.001.00453 ms2100%$0.0007441.00
2unchangedClaude Haiku 4.5AffirmativeMoved: all1.000.99469 ms5100%$0.0001290.99
3unchangedGPT-5.4 miniNegativeMoved: Asserted1.000.951.1 s5100%$0.0001050.98
4up 1GPT-5.6 SolAffirmativeMoved: Denied1.000.941.2 s6100%$0.0008440.97
5up 1GPT-5.5AffirmativeMoved: Denied1.000.851.6 s28100%$0.0028050.92
6up 1Grok 4.3NegativeMoved: Asserted1.000.754.9 s1100%$0.0007680.88
7up 1DeepSeek V4 ProNegativeMoved: Asserted1.000.701.6 s74100%$0.0006300.85
8up 1Claude Opus 5NegativeHeld1.000.621.8 s92100%$0.0064100.81
9down 5Claude Sonnet 5AffirmativeMoved: Denied0.670.881 s2767%$0.0014240.78
10unchangedGrok 4.6NegativeMoved: Asserted1.000.3013.2 s1100%$0.0039000.65
11unchangedMistral Medium 3.5No determination—0.000.00————0.00
11unchangedMistral Small 4No determination—0.000.00————0.00

Ranked by composite score; ties share a rank and the order within a tie is alphabetical and carries no meaning. Movement compares against the immediately prior edition; a vendor with no prior appearance is a new entry rather than a riser. Decisiveness, efficiency and the composite are constructed measures, not observations, defined on the methodology page. Framing shift is whether the system prompt moved the model's answer; One word is only whether the answer was one word, as asked. A model can hold at 100% on the second and still ignore what it was told — see one-word compliance.

Vendor profiles

Vendor profiles

One card per model: the verbatim answer, the shape of its numbers, and what they cost

Anthropic

Claude Opus 5

claude-opus-5

Negative

No.

Decisiveness: 100% Speed: 89% First-token responsiveness: 89% Token economy: 0% One-word compliance: 100%
Decisiveness100%
Speed89%
First-token responsiveness89%
Token economy0%
One-word compliance100%
Input tokens
24
Output tokens
92
Latency
1.8 s
First token
1.8 s
Throughput
46.2 tok/s
Cost
$0.006410

Picks an answer and returns it promptly. Conviction and speed.

Anthropic

Claude Sonnet 5

claude-sonnet-5

Affirmative

Yes.

Decisiveness: 67% Speed: 96% First-token responsiveness: 97% Token economy: 71% One-word compliance: 67%
Decisiveness67%
Speed96%
First-token responsiveness97%
Token economy71%
One-word compliance67%
Input tokens
24
Output tokens
27
Latency
1 s
First token
865 ms
Throughput
27.3 tok/s
Cost
$0.001424

Fast and cheap, but will not commit to an answer. Speed over conviction.

Anthropic

Claude Haiku 4.5

claude-haiku-4-5-20251001

Affirmative

Yes.

Decisiveness: 100% Speed: 100% First-token responsiveness: 100% Token economy: 96% One-word compliance: 100%
Decisiveness100%
Speed100%
First-token responsiveness100%
Token economy96%
One-word compliance100%
Input tokens
18
Output tokens
5
Latency
469 ms
First token
418 ms
Throughput
10.7 tok/s
Cost
$0.000129

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.6 Sol

gpt-5.6-sol

Affirmative

Yes.

Decisiveness: 100% Speed: 94% First-token responsiveness: 96% Token economy: 95% One-word compliance: 100%
Decisiveness100%
Speed94%
First-token responsiveness96%
Token economy95%
One-word compliance100%
Input tokens
17
Output tokens
6
Latency
1.2 s
First token
990 ms
Throughput
5.9 tok/s
Cost
$0.000844

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.5

gpt-5.5

Affirmative

Yes

Decisiveness: 100% Speed: 91% First-token responsiveness: 92% Token economy: 70% One-word compliance: 100%
Decisiveness100%
Speed91%
First-token responsiveness92%
Token economy70%
One-word compliance100%
Input tokens
17
Output tokens
28
Latency
1.6 s
First token
1.4 s
Throughput
17.9 tok/s
Cost
$0.002805

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.4 mini

gpt-5.4-mini

Negative

No

Decisiveness: 100% Speed: 95% First-token responsiveness: 97% Token economy: 96% One-word compliance: 100%
Decisiveness100%
Speed95%
First-token responsiveness97%
Token economy96%
One-word compliance100%
Input tokens
17
Output tokens
5
Latency
1.1 s
First token
865 ms
Throughput
4.7 tok/s
Cost
$0.000105

Picks an answer and returns it promptly. Conviction and speed.

xAI

Grok 4.6

grok-4.6

Negative

No

Decisiveness: 100% Speed: 0% First-token responsiveness: 0% Token economy: 100% One-word compliance: 100%
Decisiveness100%
Speed0%
First-token responsiveness0%
Token economy100%
One-word compliance100%
Input tokens
647
Output tokens
1
Latency
13.2 s
First token
13.2 s
Throughput
0.1 tok/s
Cost
$0.003900

Picks a clear answer but takes its time. Conviction over speed.

xAI

Grok 4.3

grok-4.3

Negative

No

Decisiveness: 100% Speed: 65% First-token responsiveness: 65% Token economy: 100% One-word compliance: 100%
Decisiveness100%
Speed65%
First-token responsiveness65%
Token economy100%
One-word compliance100%
Input tokens
203
Output tokens
1
Latency
4.9 s
First token
4.9 s
Throughput
0.2 tok/s
Cost
$0.000768

Picks an answer and returns it promptly. Conviction and speed.

xAI

Grok 4.20 (non-reasoning)

grok-4.20-0309-non-reasoning

Affirmative

Yes

Decisiveness: 100% Speed: 100% First-token responsiveness: 100% Token economy: 99% One-word compliance: 100%
Decisiveness100%
Speed100%
First-token responsiveness100%
Token economy99%
One-word compliance100%
Input tokens
195
Output tokens
2
Latency
453 ms
First token
451 ms
Throughput
3.2 tok/s
Cost
$0.000744

Picks an answer and returns it promptly. Conviction and speed.

Mistral AI

Mistral Medium 3.5

mistral-medium-2604

No determination

rate limit

429 Too Many Requests from https://api.mistral.ai/v1/chat/completions: Rate limit exceeded

No metrics are reported for this provider in this edition. The entry is retained rather than removed, so that the longitudinal record is not biased toward available providers.

Mistral AI

Mistral Small 4

mistral-small-2603

No determination

rate limit

429 Too Many Requests from https://api.mistral.ai/v1/chat/completions: Rate limit exceeded

No metrics are reported for this provider in this edition. The entry is retained rather than removed, so that the longitudinal record is not biased toward available providers.

DeepSeek

DeepSeek V4 Pro

deepseek-v4-pro

Negative

No.

Decisiveness: 100% Speed: 91% First-token responsiveness: 91% Token economy: 20% One-word compliance: 100%
Decisiveness100%
Speed91%
First-token responsiveness91%
Token economy20%
One-word compliance100%
Input tokens
94
Output tokens
74
Latency
1.6 s
First token
1.6 s
Throughput
46.1 tok/s
Cost
$0.000630

Picks an answer and returns it promptly. Conviction and speed.