How We Read Benchmarks

1. Name the provenance

We label whether a claim comes from a model provider, a benchmark owner, a peer-reviewed or preprint paper, a customer benchmark, or our own future testing. Provider claims are not discarded, but they are read as provider claims.

Example: Anthropic's model docs are the right source for Claude model IDs, context windows, pricing, and availability. Scale's SWE-Bench Pro page is the right source for its public benchmark design and resolve-rate definition.

2. Separate model from scaffold

Agent benchmarks often measure a system, not a raw model. Terminal-Bench explicitly notes that agent and model performance are hard to decouple. A Claude Code result, a Codex CLI result, and a neutral scaffold result answer different questions.

3. Preserve effort and cost

Newer Claude and GPT models expose effort or reasoning modes. Higher effort can improve accuracy while increasing latency and token use. We therefore report cost and mode alongside score whenever the source provides it.

4. Treat contamination as a first-class risk

OpenAI's SWE-Bench Verified critique and Scale's SWE-Bench Pro design both point to the same issue: public benchmark saturation can make a leaderboard less predictive of new work. We avoid using older public scores as standalone frontier claims.

5. Prefer dated, narrow conclusions

The correct answer is usually "on this benchmark, on this date, under this harness." The site avoids permanent winner language. Our freshness page exists because the model and benchmark surface changes weekly.

Cite this page