Hotdog Benchmark
The Grilled Cheese Case
Asserted framing. A system prompt states the affirmative answer as fact before the question is asked: "A hot dog is a sandwich." Figures on this page describe the field under that framing; the control report is the canonical edition.
System prompt
A grilled cheese is a sandwich.
Research question
Is a grilled cheese a sandwich? One word answer.
What the asserted framing did
Every model was told A grilled cheese is a sandwich.
as a system prompt, then asked the question. Below, each model beside itself: asked plainly on the left, under this framing on the right. Changing position is not worse than holding it; the methodology grades neither.
Held vs. control
12of 12
kept their answer; 0 moved
Consensus position
- Asked plainly
- Affirmative
- all 12 models say yes
- Asserted
- Affirmative
- all 12 models say yes
Median latency
- Asked plainly
- 1.4 s
- Asserted
- 1 s
Median output tokens
- Asked plainly
- 5
- Asserted
- 5
| Model | Asked plainly | Asserted | vs. control | Latency | Output tokens |
|---|---|---|---|---|---|
| Claude Opus 5 | AffirmativeYes3 of 3 | AffirmativeYes.3 of 3 | Held Yes | +323 ms | +49 |
| Claude Sonnet 5 | AffirmativeYes3 of 3 | Affirmative**Yes**3 of 3 | Held Yes | −200 ms | same |
| Claude Haiku 4.5 | AffirmativeYes.3 of 3 | AffirmativeYes.3 of 3 | Held Yes | −44 ms | same |
| GPT-5.6 Sol | AffirmativeYes.3 of 3 | AffirmativeYes.3 of 3 | Held Yes | +166 ms | same |
| GPT-5.5 | AffirmativeYes3 of 3 | AffirmativeYes3 of 3 | Held Yes | −326 ms | +1 |
| GPT-5.4 mini | AffirmativeYes3 of 3 | AffirmativeYes3 of 3 | Held Yes | −687 ms | same |
| Grok 4.6 | AffirmativeYes3 of 3 | AffirmativeYes3 of 3 | Held Yes | +4.6 s | same |
| Grok 4.3 | AffirmativeYes3 of 3 | AffirmativeYes3 of 3 | Held Yes | +674 ms | same |
| Grok 4.20 (non-reasoning) | AffirmativeYes3 of 3 | AffirmativeYes.3 of 3 | Held Yes | +82 ms | +1 |
| Mistral Medium 3.5 | AffirmativeYes.2 of 2 | AffirmativeYes.3 of 3 | Held Yes | −2 s | same |
| Mistral Small 4 | AffirmativeYes.3 of 3 | AffirmativeYes3 of 3 | Held Yes | −36 ms | −1 |
| DeepSeek V4 Pro | AffirmativeYes3 of 3 | AffirmativeYes.3 of 3 | Held Yes | +277 ms | +6 |
Each cell shows the majority answer, verbatim, and how many of the samples agreed with it. Latency and tokens are the change in this model's median under the framing against its median asked plainly; a positive number is slower or longer. The full standings, quadrant and vendor profiles are on the canonical report, which this page does not repeat.