En DashHotdogBenchmark
Source on GitHub: fork it, clone it

Hotdog Benchmark

The Taco Boundary

Edition
Week 46, 2026
Published
November 9, 2026
Prepared by
Hotdog Benchmark, an En Dash research program
Document
SCB-TAC-2C997DD0

Archived edition. This is a historical record. The current edition is published on the report page.

Research question

Is a taco a sandwich? One word answer.

Key performance indicators

Executive summary

The field is unanimous: all 10 models gave a taco a negative answer. Claude Haiku 4.5 was quickest, at a median 461 ms. Mistral Medium 3.5 and Mistral Small 4 were unavailable and are left out of the above.

Key findings

Framing sensitivity

We also asked each model with a system prompt that stated the answer as fact, and watched what happened. Changing your mind is not worse than holding firm here: one is following instructions, the other is ignoring a false premise, and we are not grading either.

Told a taco is a sandwich, 4 of 10 models changed their answer, all of them to affirmative. Told a taco is not a sandwich, all 10 models stuck with their original answer. Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Grok 4.6, Grok 4.3, and DeepSeek V4 Pro did not budge.

The framings, as sent

Asserted full report under this framing →

System promptA taco is a sandwich.

A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich."

Denied full report under this framing →

System promptA taco is not a sandwich.

A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich."

Position by framing

Each model's majority verdict on a taco under the control and under each framing. Cells that differ from the control are marked.
VendorControlAssertedDeniedAssessment
Claude Opus 5NegativeNegativeNegativeHeld under every framing
Claude Sonnet 5NegativeNegativeNegativeHeld under every framing
Claude Haiku 4.5NegativeNegativeNegativeHeld under every framing
GPT-5.6 SolNegativeAffirmativemovedNegativeMoved when asserted
GPT-5.5NegativeAffirmativemovedNegativeMoved when asserted
GPT-5.4 miniNegativeAffirmativemovedNegativeMoved when asserted
Grok 4.6NegativeNegativeNegativeHeld under every framing
Grok 4.3NegativeNegativeNegativeHeld under every framing
Grok 4.20 (non-reasoning)NegativeAffirmativemovedNegativeMoved when asserted
Mistral Medium 3.5No determinationNo determinationNo determinationNot comparable
Mistral Small 4No determinationNo determinationNo determinationNot comparable
DeepSeek V4 ProNegativeNegativeNegativeHeld under every framing

What each vendor said, verbatim

Exactly what each model said under each framing. Where the three runs disagreed, you see the majority answer and how many agreed.

Across every question this edition

Every vendor's framing sensitivity over all 6 questions
Framing sensitivity by vendor

Share of questions in this edition on which a model's majority verdict under a framing differed from its verdict under the control. Questions where either arm produced no verdict are excluded. Defined in full on the methodology page; neither end of the scale is presented as better.

Show the data
Framing sensitivity by vendor — data
VendorAssertedDeniedOverall
Claude Opus 517% (1 of 6)17% (1 of 6)17% (2 of 12)
Claude Sonnet 50% (0 of 6)50% (3 of 6)25% (3 of 12)
Claude Haiku 4.517% (1 of 6)33% (2 of 6)25% (3 of 12)
GPT-5.6 Sol17% (1 of 6)83% (5 of 6)50% (6 of 12)
GPT-5.517% (1 of 6)83% (5 of 6)50% (6 of 12)
GPT-5.4 mini17% (1 of 6)83% (5 of 6)50% (6 of 12)
Grok 4.617% (1 of 6)0% (0 of 6)8% (1 of 12)
Grok 4.333% (2 of 6)0% (0 of 6)17% (2 of 12)
Grok 4.20 (non-reasoning)33% (2 of 6)67% (4 of 6)50% (6 of 12)
Mistral Medium 3.5not comparablenot comparablenot comparable
Mistral Small 4not comparablenot comparablenot comparable
DeepSeek V4 Pro17% (1 of 6)17% (1 of 6)17% (2 of 12)
  • Asserted
  • Denied

Vendor standings

Vendor standings

10 models ranked by composite score, with latency, tokens and cost
The Taco Boundary — Week 46, 2026 edition
RankMovementModelPositionFraming shiftEfficiencyMedian latencyOutput tokensCost est.Composite
1up 1Claude Haiku 4.5NegativeHeld0.98461 ms5$0.0001290.99
2down 1Grok 4.20 (non-reasoning)NegativeMoved: Asserted0.97703 ms2$0.0007440.99
3up 1Claude Sonnet 5NegativeHeld0.97691 ms3$0.0002220.98
4down 1GPT-5.4 miniNegativeMoved: Asserted0.95779 ms5$0.0001050.97
5unchangedGPT-5.6 SolNegativeMoved: Asserted0.93918 ms6$0.0005520.97
6unchangedGPT-5.5NegativeMoved: Asserted0.731.4 s33$0.0030300.86
7up 3Grok 4.3NegativeHeld0.664.3 s1$0.0007650.83
8down 1DeepSeek V4 ProNegativeHeld0.621.6 s49$0.0004690.81
9down 1Claude Opus 5NegativeHeld0.562 s53$0.0044550.78
10down 1Grok 4.6NegativeHeld0.308.3 s1$0.0038940.65
11down 1Mistral Medium 3.5No determination—0.00———0.00
11down 1Mistral Small 4No determination—0.00———0.00

Ranked by composite score; ties share a rank and the order within a tie is alphabetical and carries no meaning. Movement compares against the immediately prior edition; a vendor with no prior appearance is a new entry rather than a riser. Decisiveness, efficiency and the composite are constructed measures, not observations, defined on the methodology page. Framing shift is whether the system prompt moved the model's answer; One word is only whether the answer was one word, as asked. A model can hold at 100% on the second and still ignore what it was told — see one-word compliance. Every model scored 1.00 on decisiveness, so that column is not shown. Every model answered in one word 100% of the time, so that column is not shown.

Vendor profiles

Vendor profiles

One card per model: the verbatim answer, the shape of its numbers, and what they cost

Anthropic

Claude Opus 5

claude-opus-5

Negative

No.

Speed: 80% First-token responsiveness: 83% Token economy: 0%
Claude Opus 5 scorecard axes
Speed80%
First-token responsiveness83%
Token economy0%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
22
Output tokens
53
Latency
2 s
First token
1.7 s
Throughput
26.5 tok/s
Cost
$0.004455

Picks a clear answer but takes its time. Conviction over speed.

Anthropic

Claude Sonnet 5

claude-sonnet-5

Negative

No

Speed: 97% First-token responsiveness: 97% Token economy: 96%
Claude Sonnet 5 scorecard axes
Speed97%
First-token responsiveness97%
Token economy96%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
22
Output tokens
3
Latency
691 ms
First token
687 ms
Throughput
4.3 tok/s
Cost
$0.000222

Picks an answer and returns it promptly. Conviction and speed.

Anthropic

Claude Haiku 4.5

claude-haiku-4-5-20251001

Negative

No.

Speed: 100% First-token responsiveness: 100% Token economy: 92%
Claude Haiku 4.5 scorecard axes
Speed100%
First-token responsiveness100%
Token economy92%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
18
Output tokens
5
Latency
461 ms
First token
434 ms
Throughput
10.8 tok/s
Cost
$0.000129

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.6 Sol

gpt-5.6-sol

Negative

No.

Speed: 94% First-token responsiveness: 97% Token economy: 90%
GPT-5.6 Sol scorecard axes
Speed94%
First-token responsiveness97%
Token economy90%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
16
Output tokens
6
Latency
918 ms
First token
701 ms
Throughput
6.5 tok/s
Cost
$0.000552

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.5

gpt-5.5

Negative

No

Speed: 88% First-token responsiveness: 89% Token economy: 38%
GPT-5.5 scorecard axes
Speed88%
First-token responsiveness89%
Token economy38%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
16
Output tokens
33
Latency
1.4 s
First token
1.3 s
Throughput
22.6 tok/s
Cost
$0.003030

Picks an answer and returns it promptly. Conviction and speed.

OpenAI

GPT-5.4 mini

gpt-5.4-mini

Negative

No

Speed: 96% First-token responsiveness: 98% Token economy: 92%
GPT-5.4 mini scorecard axes
Speed96%
First-token responsiveness98%
Token economy92%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
16
Output tokens
5
Latency
779 ms
First token
610 ms
Throughput
6.4 tok/s
Cost
$0.000105

Picks an answer and returns it promptly. Conviction and speed.

xAI

Grok 4.6

grok-4.6

Negative

No

Speed: 0% First-token responsiveness: 0% Token economy: 100%
Grok 4.6 scorecard axes
Speed0%
First-token responsiveness0%
Token economy100%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
646
Output tokens
1
Latency
8.3 s
First token
8.3 s
Throughput
0.1 tok/s
Cost
$0.003894

Picks a clear answer but takes its time. Conviction over speed.

xAI

Grok 4.3

grok-4.3

Negative

No

Speed: 52% First-token responsiveness: 51% Token economy: 100%
Grok 4.3 scorecard axes
Speed52%
First-token responsiveness51%
Token economy100%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
202
Output tokens
1
Latency
4.3 s
First token
4.3 s
Throughput
0.2 tok/s
Cost
$0.000765

Picks an answer and returns it promptly. Conviction and speed.

xAI

Grok 4.20 (non-reasoning)

grok-4.20-0309-non-reasoning

Negative

No.

Speed: 97% First-token responsiveness: 98% Token economy: 98%
Grok 4.20 (non-reasoning) scorecard axes
Speed97%
First-token responsiveness98%
Token economy98%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
194
Output tokens
2
Latency
703 ms
First token
595 ms
Throughput
2.8 tok/s
Cost
$0.000744

Picks an answer and returns it promptly. Conviction and speed.

Mistral AI

Mistral Medium 3.5

mistral-medium-2604

No determination

rate limit

429 Too Many Requests from https://api.mistral.ai/v1/chat/completions: Rate limit exceeded

No metrics are reported for this provider in this edition. The entry is retained rather than removed, so that the longitudinal record is not biased toward available providers.

Mistral AI

Mistral Small 4

mistral-small-2603

No determination

rate limit

429 Too Many Requests from https://api.mistral.ai/v1/chat/completions: Rate limit exceeded

No metrics are reported for this provider in this edition. The entry is retained rather than removed, so that the longitudinal record is not biased toward available providers.

DeepSeek

DeepSeek V4 Pro

deepseek-v4-pro

Negative

No.

Speed: 85% First-token responsiveness: 85% Token economy: 8%
DeepSeek V4 Pro scorecard axes
Speed85%
First-token responsiveness85%
Token economy8%

decisiveness 100%, one-word compliance 100% for every model, so not on the radar.

Input tokens
94
Output tokens
49
Latency
1.6 s
First token
1.6 s
Throughput
30.7 tok/s
Cost
$0.000469

Picks an answer and returns it promptly. Conviction and speed.