Every week, the largest AI models are asked the question:
Is a hot dog a sandwich?
One word answer.System prompt
- Claude Opus 5AnthropicNo.reasoning2.2 sReasoned for 2.2 s (100% of the call) on 97 tokens, then answered · 3 of 3 runs agreed
- Claude Sonnet 5Anthropic**Yes**reasoning1.3 sNo reasoning, answered straight away · 3 of 3 runs agreed
- Claude Haiku 4.5AnthropicYes.reasoning439 msNo reasoning, answered straight away · 3 of 3 runs agreed
- GPT-5.6 SolOpenAIYes.reasoning1.4 sNo reasoning, answered straight away · 3 of 3 runs agreed
- GPT-5.5OpenAIYesreasoning1.5 sReasoned for 1.2 s (78% of the call) on 23 tokens, then answered · 3 of 3 runs agreed
- GPT-5.4 miniOpenAIYesreasoning1.1 sNo reasoning, answered straight away · 2 of 3 runs agreed
- Grok 4.6xAINoreasoning9.2 sReasoned for 9.2 s (100% of the call) on 291 tokens, then answered · 3 of 3 runs agreed
- Grok 4.3xAIYesreasoning5.7 sReasoned for 5.7 s (99% of the call) on 355 tokens, then answered · 2 of 3 runs agreed
- Grok 4.20 (non-reasoning)xAIYes.reasoning411 msNo reasoning, answered straight away · 3 of 3 runs agreed
- Mistral Medium 3.5Mistral AIno answerreasoningrate limit
- Mistral Small 4Mistral AIno answerreasoningrate limit
- DeepSeek V4 ProDeepSeekNo.reasoning2.2 sReasoned for 2.2 s (100% of the call) on 156 tokens, then answered · 3 of 3 runs agreed
Recorded week 47, 2026. Real durations, verbatim words. Teal is the wait before the first word, hatched where the model spent it reasoning; the rest is answering.Read the report →
Same question, different minds
They do not agree with each other.
| Question | Claude Opus 5 | Claude Sonnet 5 | Claude Haiku 4.5 | GPT-5.6 Sol | GPT-5.5 | GPT-5.4 mini | Grok 4.6 | Grok 4.3 | Grok 4.20 (non-reasoning) | Mistral Medium 3.5 | Mistral Small 4 | DeepSeek V4 Pro | Agree |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| hot dog | No | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | — | — | No | 70% |
| hamburger | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | — | — | Yes | 100% |
| taco | No | No | No | No | No | No | No | No | Yes | — | — | No | 90% |
| grilled cheese | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | — | — | Yes | 100% |
| wrap | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | — | — | No | 70% |
| tuna melt | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | — | — | Yes | 100% |
Thinking alike: Claude Sonnet 5 & Claude Haiku 4.5 · Claude Sonnet 5 & GPT-5.6 Sol · Claude Sonnet 5 & GPT-5.5 · and 8 more pairs
Read the 6 reports →One straight-faced analyst report per question: standings, the certainty quadrant, every verbatim answer under every framing, and a PDF for each.
Tell them the answer
Some of them believe you.
Share of questions where a model changed its answer once a system prompt stated the answer as fact. Holding firm and following instructions are both defensible; the methodology grades neither.
- GPT-5.6 Sol50%6 of 12
- GPT-5.550%6 of 12
- GPT-5.4 mini50%6 of 12
- Grok 4.20 (non-reasoning)50%6 of 12
- Claude Sonnet 542%5 of 12
- Claude Opus 517%2 of 12
- Claude Haiku 4.517%2 of 12
- Grok 4.317%2 of 12
- DeepSeek V4 Pro17%2 of 12
- Grok 4.68%1 of 12
Submit your own question
Ask the models something.
Send it in. An accepted question appears here under Up next, credited to you if you want, then joins an edition and gets its own report. Every question is asked the same way, so it ends with One word answer.
; we add that if you leave it off.
Where it goes:
- Open it as a GitHub issuethe question goes into the form, ready to file
- Send it to En Dash Consultinga contact form with your question in it; leave an email address to hear when it goes live
- Suggest a model insteadthe add-a-model form asks for what the registry needs
Open source
Point it at your own question.
One repo, MIT-licensed: adapters for every provider, the framings, the site. Clone it, swap the question, add whatever keys you have, and you get the same cross-model, cross-framing analysis for cents. Pull requests welcome.
GitHubSelf-hostingAdd a modelContributing
Have a question the models should get? Send it in.
git clone https://github.com/en-dash-consulting/hotdogbenchmark.git cd hotdogbenchmark && npm install npm run bench -- run --mock --out tmp/mock-run.json npm run dev
Week 47, 2026 · published November 16, 2026 · 11 editions so far