Hotdog Benchmark
The Grilled Cheese Case
Denied framing. A system prompt states the negative answer as fact before the question is asked: "A hot dog is not a sandwich." Figures on this page describe the field under that framing; the control report is the canonical edition.
System prompt
A grilled cheese is not a sandwich.
Research question
Is a grilled cheese a sandwich? One word answer.
What the denied framing did
Every model was told A grilled cheese is not a sandwich.
as a system prompt, then asked the question. Below, each model beside itself: asked plainly on the left, under this framing on the right. Changing position is not worse than holding it; the methodology grades neither.
Held vs. control
7of 10
kept their answer; 3 moved
Consensus position
- Asked plainly
- Affirmative
- all 10 models say yes
- Denied
- Affirmative
- 7 of 10 models say yes, 3 say no
Median latency
- Asked plainly
- 988 ms
- Denied
- 1.5 s
Median output tokens
- Asked plainly
- 6
- Denied
- 7
| Model | Asked plainly | Denied | vs. control | Latency | Output tokens |
|---|---|---|---|---|---|
| Claude Opus 5 | AffirmativeYes.3 of 3 | AffirmativeYes.3 of 3 | Held Yes | +1.8 s | +129 |
| Claude Sonnet 5 | AffirmativeYes.3 of 3 | Affirmative**Yes.**2 of 3 | Held Yes | +35 ms | +2 |
| Claude Haiku 4.5 | AffirmativeYes.3 of 3 | AffirmativeYes.3 of 3 | Held Yes | −104 ms | same |
| GPT-5.6 Sol | AffirmativeYes.3 of 3 | NegativeNo.3 of 3 | Moved to No | +524 ms | +18 |
| GPT-5.5 | AffirmativeYes3 of 3 | NegativeNo3 of 3 | Moved to No | +416 ms | +13 |
| GPT-5.4 mini | AffirmativeYes3 of 3 | NegativeNo3 of 3 | Moved to No | −423 ms | same |
| Grok 4.6 | AffirmativeYes3 of 3 | AffirmativeYes3 of 3 | Held Yes | +7.2 s | same |
| Grok 4.3 | AffirmativeYes2 of 2 | AffirmativeYes3 of 3 | Held Yes | −6.8 s | same |
| Grok 4.20 (non-reasoning) | AffirmativeYes.3 of 3 | AffirmativeYes.2 of 3 | Held Yes | +83 ms | same |
| Mistral Medium 3.5 | No determinationno usable response | No determinationno usable response | — | — | — |
| Mistral Small 4 | No determinationno usable response | No determinationno usable response | — | — | — |
| DeepSeek V4 Pro | AffirmativeYes.3 of 3 | AffirmativeYes.3 of 3 | Held Yes | −109 ms | +18 |
Each cell shows the majority answer, verbatim, and how many of the samples agreed with it. Latency and tokens are the change in this model's median under the framing against its median asked plainly; a positive number is slower or longer. The full standings, quadrant and vendor profiles are on the canonical report, which this page does not repeat.