En DashHotdogBenchmark

Add a model

4 providers are currently evaluated. Adding a model to one of them is a one-line data change. Adding a new provider is one new file.

You do not need an API key to work on any of this — mock mode replays recorded responses, so the whole pipeline runs offline.

Adding a model to an existing provider

  1. Add an entry to models.json. Copy the model ID from the provider's documentation rather than typing it from memory, and record the page you copied it from in docsUrl.

  2. Fill in pricing with an asOf date. Every cost estimate on this site is stamped with when its rate was read; an undated price table produces numbers with no meaning.

  3. Check it against the live API:

    npm run bench:smoke -- --provider <id>

    That makes exactly one call and prints the text, the token usage and the timing. It costs a fraction of a cent.

  4. Open a pull request. CI runs lint, types, tests, schema validation and accessibility checks.

Adding a new provider

Start by reading src/providers/anthropic.ts. It is the reference adapter: short, linear, and commented for exactly this purpose. The how it works page shows the interface an adapter satisfies.

  1. Copy anthropic.ts to src/providers/<vendor>.ts and implement complete() against the vendor's API. If it speaks OpenAI's chat-completions dialect, use openai-compatible.ts instead and your adapter is about fifteen lines.

  2. Use fetchWithPolicy from http.ts rather than calling fetch directly, so your adapter gets the same timeout, retry and error classification behavior as every other.

  3. Map the vendor's usage payload onto the shared shape and add your row to usage-normalization.md, including whether reasoning tokens are counted inside output tokens for that vendor. That column is the one that changes how a chart should be read.

  4. Record fixtures, so the adapter is testable without a key:

    npm run bench:record -- --provider <id>
  5. Register it in all.ts, add its key variable to src/env.ts and .env.example, and add the model to models.json.

Two rules the linter enforces

Nothing under src/providers or src/runner may import a node: builtin or read process.env. Credentials and fetch arrive through an injected context instead.

That is what lets the same adapters run in a browser later. It costs nothing to decide up front and means touching every adapter to retrofit.

If you would rather just suggest one

Open an add-a-model issue. It asks for the provider, the exact model ID, the docs and pricing URLs, and whether the API supports streaming and usage reporting — which is everything needed to action it.

Full contributor documentation is in CONTRIBUTING.md.