How it works

1. Decide what you are asking

   Questions live in a JSON file, not in code. Adding one is a data change, and the schema enforces that every question ends with the same instruction — which is what makes the compliance metric mean anything.

   questions.json

2. Decide who you are asking

   Model IDs live here and nowhere else. Every entry records the documentation page its ID was verified against and the date its pricing was read, because both go stale and a cost estimate from an undated table is a number with no meaning.

   models.json

3. Write one adapter per vendor

   Each adapter turns one vendor’s HTTP API into the same small shape. It receives its API key and its fetch by injection, so it can be tested with no network and later run in a browser.

   src/providers/anthropic.ts

4. Share the boring parts

   Timeouts, retries with jittered backoff, and error classification are identical across every vendor, so they are written once. An adapter that called fetch directly would reimplement this badly.

   src/providers/http.ts

5. Ask everything, several times, politely

   The runner asks every model every question three times, with bounded concurrency and never more than one in-flight request per provider. That second constraint matters: three simultaneous calls to one vendor invites a rate limit, and a retried request measures your own impatience rather than the provider.

   src/runner/run.ts

6. Write the answer down, permanently

   One JSON file per ISO week, validated against a versioned schema and committed. Re-running a week corrects that edition rather than creating a second one.

   data/runs

7. Do it again next week without being asked

   A scheduled GitHub Action runs the benchmark with keys from repository secrets, commits the new data, and triggers a redeploy. Nobody runs the benchmark by hand.

   .github/workflows/benchmark.yml

export interface ProviderAdapter {
  /** Stable id matching `models.json` and the credential map. */
  readonly id: string
  /** Human name for logs and error messages. */
  readonly displayName: string
  /** Ask the model. Throws {@link ProviderError} on any failure. */
  complete(request: CompleteRequest, context: AdapterContext): Promise<CompleteResult>
}

From src/providers/types.ts, read from source when this page was built.

for await (const event of readSseJson<AnthropicEvent>(response)) {
  switch (event.type) {
    case 'message_start':
      // Carries input_tokens and the cache counts. Output tokens here
      // are a running start, superseded by message_delta below.
      usage = { ...usage, ...event.message?.usage }
      break

    case 'content_block_delta':
      if (event.delta?.type === 'text_delta' && event.delta.text) {
        // The first content token: this is the moment ttfbMs measures.
        // Marking is idempotent, so calling it per delta is fine.
        measurement.markFirstToken()
        context.onFirstToken?.()
        text += event.delta.text
      }
      break

    case 'message_delta':
      // Anthropic documents these counts as *cumulative*, so the last
      // message_delta holds the authoritative totals. Merging rather
      // than replacing keeps input_tokens from message_start when a
      // delta omits it.
      usage = { ...usage, ...event.usage }
      break

    case 'error':
      // An error can arrive mid-stream, after a 200. The HTTP layer has
      // already returned by then, so it has to be caught here.
      throw new ProviderError(
        'server',
        `Anthropic stream error: ${event.error?.message ?? event.error?.type ?? 'unknown'}`,
      )
  }
}

From src/providers/anthropic.ts.

npm run bench -- run --mock
npm run dev