How to Pick a Model Without Reading a Leaderboard

Ask which model is best and you get a leaderboard. Ask which model is best for this task, this budget, and this latency target, and you get a far more useful answer — usually a far more boring one too. Model selection is not a ranking problem. It is a fit problem.

Start with the task

A frontier model doing address extraction is waste. A small model attempting multi-step reasoning is a support ticket. Write down what the task actually demands — accuracy bar, reasoning depth, context length, structured output, tolerance for latency — before you look at a single model card. Requirements narrow the field. A leaderboard only widens it.

Match capability to need

A surprising amount of production work — classification, routing, extraction, summarization, tagging — runs well on small, fast, inexpensive models. Reserve the expensive ones for genuinely hard reasoning and the model bill stops being the thing finance asks about every month.

The numbers that decide it

Once a model clears your quality bar, the choice comes down to three things: cost per request at real volume, latency per request, and reliability under load. A model that is slightly smarter but noticeably slower and meaningfully more expensive is the wrong call for anything a user is sitting and waiting on. Optimize the experience, not the benchmark.

• Quality — does it pass your evals, not somebody else’s?
• Cost — what does it cost at the volume you actually expect?
• Latency — is it fast enough for the interaction you are building?
• Portability — can you switch providers without a rewrite?

Build your own evals

Public benchmarks measure general capability on generic problems. They say close to nothing about your data, your edge cases, your tone, or the specific way your customers phrase things. A short, honest eval set pulled from your real workload beats every leaderboard combined, because it tests the thing you actually ship.

Route, do not standardize

Most systems end up using more than one model anyway: a cheap one for the common path, a stronger one for the hard cases, and a rule that decides which is which. Routing costs a little engineering up front and keeps paying once traffic grows.

“The right model is the cheapest, fastest one that still passes your evals. Everything above that line is capability you are paying for and not using.”

Re-run the decision

Prices drop, models get deprecated, and a choice that was correct at launch can be beaten a quarter later. If the abstraction is clean and the evals still run, re-evaluating takes an afternoon. That optionality is worth more than any single model you could commit to today.