Fifty examples and a spreadsheet: how a small company picks an AI model for one job

A small company that wants AI to do one job, reading supplier invoices into the accounts, drafting replies to support mail, sorting enquiries by urgency, faces a choice that looks technical and is not. There are dozens of models from several providers, at prices that differ by a factor of forty or more, and every one of them tops some leaderboard. None of those leaderboards contains your invoices.

The way to choose is an afternoon of work that anyone in the company can do: collect fifty real cases, run them through two or three models, have the person who does the job today mark the answers without knowing which model wrote which, and put quality and cost side by side in a spreadsheet. This article walks through it, with the numbers for one example.

Why the leaderboard does not answer the question

Public benchmarks measure general skill on shared test sets: exam questions, coding puzzles, reasoning problems. They are useful for telling which models are broadly capable. They do not tell you whether a model reads the totals off your largest supplier's invoices correctly when the VAT line is printed above the subtotal, or whether its replies to your customers sound like your company.

OpenAI's own guide to evaluations lists "vibe-based evals", using "it seems like it's working" as the test, as the first thing to avoid, along with "waiting until you ship before implementing any evals" (OpenAI, Evaluation best practices). Anthropic's guide starts in the same place: decide what success means for your task, specifically and measurably, and design tests that "mirror your real-world task distribution" (Anthropic, Define success criteria and build evaluations). Both are written for engineers, but the idea fits a small office: test on your own work, not someone else's.

Step 1: collect fifty real cases

Fifty is not a magic number. It is enough to see the difference between a model that is right nine times in ten and one that is right six times in ten, and small enough that one person can mark the answers in an hour or two. Take them from real work in the last few months, with names and account numbers removed:

- About thirty ordinary cases, picked at random, not the ones you remember. If 70% of your invoices come from five suppliers, roughly 70% of the cases should too.
- About fifteen hard ones: handwritten notes, a scanned page photographed at an angle, a customer who asks three questions in one email, an invoice in another currency.
- About five that should be refused or flagged: an email that is spam, a document that is not an invoice at all, a request the company never handles. Anthropic's list of edge cases to include starts with "irrelevant or nonexistent input data" for a reason.
- Every case that ever went wrong. If a customer complained, it goes in.

For each case, write down what a correct answer is before you run any model. For invoices that is easy: the supplier, the date, the total, the tax. For support replies it is a short note of what the reply must say and must not say ("offers the refund; does not promise a delivery date"). This note is what the marker checks against later, and writing it first stops the first model's answer from setting the standard.

Step 2: run two or three models with the same prompt

Pick a cheap model, a middle one and an expensive one, from one provider or two. Write one prompt that explains the job the way you would explain it to a new hire, and use it unchanged for every model. Run all fifty cases through each.

You do not need software for this. A developer can do it in a short script, and most providers' web consoles let you paste a prompt and a case and copy the answer out. What matters is that every model sees the same prompt and the same cases.

Step 3: mark the answers blind

This is the step that makes the test worth doing. Put every answer in a spreadsheet, one row per answer, with the model names hidden behind letters and the rows shuffled, so the marker cannot tell that row 12 came from the expensive model.

The marker should be the person who does the job today: the bookkeeper for invoices, whoever answers the support inbox for replies. They know what a wrong answer costs, which a developer usually does not. Ask for pass or fail with a one-line reason rather than a score out of ten; OpenAI's guide recommends "pairwise comparison or pass/fail for more reliability", and a yes or no is quicker to decide and easier to agree on.

Two rules keep the marking honest. Mark against the note written in step 1, not against the other answers. And reveal the model column only when every row is marked.

Later, when the test is re-run often, it is tempting to have another model do the marking. That can work, but only after you have checked that its marks agree with your bookkeeper's on these fifty cases. Model judges are known to prefer longer answers and to be swayed by the order answers appear in; OpenAI names "position bias" and "verbosity bias" in the same guide.

Step 4: put quality and cost on one line

Now count passes per model and work out what each case costs. The cost of a case is the model's price for the words in and out, which is small, plus the cost of the mistakes, which usually is not.

Here is an example for invoice reading. The prices are Anthropic's list prices on 15 October 2026 (Anthropic, Pricing): per million tokens, Claude Haiku 5.5 is $0.10 in and $0.50 out, Claude Sonnet 5.5 is $2 in and $10 out, Claude Opus 5.5 is $4 in and $20 out. Assume an invoice is about 2,000 tokens in and the answer about 300 tokens out, and that a wrong total caught later takes the bookkeeper five minutes at $30 an hour, $2.50. The pass counts are invented for the example; yours will differ, which is the point of running the test.

The model with the lowest price is the most expensive choice here, because the errors cost a thousand times more than the model does. Running all fifty cases through all three models cost about a dollar.

The same arithmetic can point the other way. For first drafts of support replies that a person reads and edits before sending anyway, a failed draft costs a minute of rewriting, and a cheap model that passes 80% may be the right answer. For sorting enquiries into folders, where a mistake costs almost nothing, the cheapest model that passes most cases wins. The spreadsheet makes the trade visible instead of leaving it to whoever chose the model first.

Two refinements are worth adding to the sheet. First, look at the failures, not just the count: three failures that are all the same supplier's layout can often be fixed with one line in the prompt, and then the cheap model may pass. Second, check whether the provider offers a discount for work that does not need an instant answer; Anthropic's batch processing, for example, charges 50% less for requests that can wait, which suits a nightly run over the day's invoices.

Step 5: decide, and write the decision down

Choose the model with the lowest total cost per case among those whose pass rate the job can live with. Some jobs set a floor regardless of cost: if a wrong answer reaches a customer directly, you may decide nothing under 95% ships, and if no model clears it, the answer is to keep a person checking every case for now.

Write a short note next to the spreadsheet: which model, which prompt version, the pass counts, the date and why. Six months from now someone will ask why the company pays for the expensive one, and the answer will be on one page.

When to run it again

The fifty cases are now an asset. Running them again takes an hour of marking, and it answers questions that otherwise turn into arguments:

- When the provider retires your model. Hosted models are switched off on as little as two months' notice, and the replacement does not answer the same way. The same fifty cases become the regression check for the migration, as we described in what a deprecation notice costs a small product.
- When you change the prompt. Every change that fixes one case can break another.
- When a cheaper or better model comes out. Prices on the pricing page above have dropped between model generations; the test tells you whether the new cheap model is now good enough for your job.
- When a case goes wrong in real use. Add it to the set, so the set grows into a record of every mistake the job has seen.
- Every few months, regardless. OpenAI's guide calls evaluation "a continuous process"; for a small company, a calendar reminder each quarter is enough.

Anthropic's guide makes a further point that is worth growing into: "more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals". For a company starting out, fifty hand-marked cases are the foundation. Once the job is in production, it pays to add cases that can be checked automatically, like totals that must match to the cent, so that hundreds can run without anyone marking them.

What it does not tell you

A test of fifty cases says how a model does on your work today. It does not say how the provider will treat your data, how often its service is down, or what its contract allows; those are questions for the terms of service and, for larger customers, the contract. And it does not replace a decision about whether the job should use AI at all. If the test shows that every model fails a third of the cases, that is a useful result too: it has cost an afternoon and about a dollar, which is much less than finding out from customers.

Sources

OpenAI, Evaluation best practices (vibe-based evals, production data, human labellers, pass/fail and pairwise comparison, position and verbosity bias in model judges, continuous evaluation); Anthropic, Define success criteria and build evaluations (specific and measurable success criteria, task-specific tests, edge cases, volume over hand-graded quality); Anthropic, Pricing (list prices per million tokens and the 50% batch discount). Pages read on 15 October 2026; prices change, so check the current page before you do the arithmetic. The pass counts, token sizes, error cost and monthly totals in the example are illustrations of our own, not measurements.