A faster model at the same price: what to check before you switch
Claude Sonnet 5.5 and GPT-6.1 Sol arrived at their predecessors' prices. Here's a small, repeatable way to decide whether a model upgrade is safe for your workflows.
Last checked:
SectionsTwo of the models most businesses actually run their workflows on were replaced this week, and neither one got more expensive.
Anthropic released Claude Sonnet 5.5 on September 28. It says outputs are more than 30% faster than Sonnet 5, at the same price: $2 per million input tokens and $10 per million output tokens. OpenAI followed within days with GPT-6.1 Sol, also at its predecessor's price, with cached input cut in half.
If you have an inbox sorter, a document extractor, or a customer reply drafter running on either family, the obvious question is whether to switch. My answer is usually yes, after a short test. This post is about that test: what to save, what to compare, and what to do when the new model is better at most things and worse at one that matters.
What actually changed
Here's the short version of both releases, with the vendor's own pages linked. The claims are theirs. I haven't benchmarked either model for this post.
Claude Sonnet 5.5. Anthropic's announcement lists three things a business owner would care about:
- Output is "30%+ faster than Sonnet 5."
- Input and output prices are unchanged. Reading from the prompt cache costs $0.10 per million tokens, half of what Sonnet 5 charged.
- In Anthropic's testing it "costs up to 30% less per task than its predecessor," because it tends to finish the same work in fewer tokens.
Anthropic is also direct about the limit: Opus 5.5 is still "clearly stronger" on complex, open-ended work that needs sustained judgment.
GPT-6.1 Sol. OpenAI's developer announcement keeps GPT-6 Sol's prices, $2 in and $10 out per million tokens, and drops cached input from $0.20 to $0.10. OpenAI reports better coding and computer-use results at a lower cost per task.
Both releases follow the same pattern. The price on the rate card stays put, and the cost of a finished task goes down because the model is quicker and wastes fewer tokens. That's good news, and it's also the part you can't confirm by reading a rate card.
Why "same price" doesn't tell you your bill
You pay per token. What you care about is cost per completed task, and three things sit between those two numbers.
Tokens used per task. A model that reasons longer, writes longer answers, or calls a tool twice can cost more on the same rate card. A model that gets to the answer sooner costs less. The vendor's "up to 30% less per task" is an average over their tests. Yours might be 40% less or 5% more.
The effort setting. Sonnet 5.5 has five effort levels, from Low to Max. Lower settings answer faster with fewer tokens, and higher ones reason longer and check their work more. The default on Anthropic's developer platform is High, while the Claude apps default to Medium. If your workflow never set this explicitly, an upgrade can change how hard the model works on every request.
Retries and corrections. If the new model gets a field wrong more often, someone fixes it by hand or the workflow runs again. That cost never shows up on the model invoice.
Here's some rough math, using made-up numbers. Say a workflow reads 3,000 customer emails a month. Each one uses about 4,000 input tokens and 500 output tokens. At $2 and $10 per million tokens, that's about $39 a month. A 30% saving is roughly $12.
That's not why you'd switch. You'd switch because replies come back faster and fewer of them need a person to fix them. If one in fifty emails gets routed to the wrong team today and the new model makes that one in a hundred, the saved staff time is worth far more than $12. If it makes that one in twenty-five, the "free" upgrade just got expensive.
So the test has to measure correct results, not token prices.
Save your test cases before you touch anything
The whole method depends on one habit: keep a small set of real examples with the answer you'd accept for each.
For most small workflows, 20 to 30 cases is enough to start. Pull them from real traffic and make sure the set includes:
- Ordinary cases. The requests you see every day. Most of the set.
- The ones that went wrong. Every time the workflow has made a mistake that someone noticed, that input belongs in the set, with the correct answer written down.
- Awkward inputs. A forwarded email chain, a photo of a handwritten form, a message in Spanish, a customer who asks two unrelated things at once.
- Cases that should be refused or escalated. A request the workflow isn't allowed to handle, or one that's missing something essential. The right result is "send this to a person."
For each case, write down what "right" means in a way you can check. "Category is Billing" is checkable. "The summary is good" is not. If a result needs judgment, write two or three specific things it must contain and one thing it must not, such as a promised appointment time the system never confirmed.
Strip or mask customer details you don't need for the test, and keep the file somewhere with the same access controls as the source data.
If you've never done this, the first set takes an afternoon. After that it's a few minutes whenever something goes wrong: add the case, add the right answer, move on. My guide to LangSmith and the rest of the eval stack covers the developer tooling for larger sets. For a small business workflow, a spreadsheet is a fine place to begin.
Run both models on the same set
Now the comparison is mechanical. Run every case through the current model and the candidate, with the same instructions and the same settings, and record four things for each run:
- Did it produce an acceptable result? Yes or no, by the rule you wrote down.
- How many tokens did it use? Input and output, from the provider's usage numbers.
- How long did it take? Wall-clock time for the whole task, tool calls included.
- Did it need a retry? A malformed response that had to be re-run counts against the model.
Run each case more than once. Models don't give identical answers every time, and a case that passes four runs out of five is a different problem from one that always fails.
Then put the results side by side. These numbers are invented to show the shape, not measured.
The line people skip is "cases that got worse." A candidate can fix three failures and introduce one new one, and the totals will hide it. Look at every case whose result changed, in either direction.
Decide with rules you wrote in advance
It's easy to talk yourself into a switch after you've seen a faster response. Write the rules down first. These are the ones I'd use for most business workflows:
- Switch when the candidate is at least as accurate overall, nothing important got worse, and cost or speed improved.
- Fix, then switch when one or two cases got worse and the cause is something you control, such as an instruction the old model happened to tolerate. Adjust the instructions, re-run the full set, and check that you didn't break anything else.
- Hold when a case that got worse is one of the expensive mistakes: money, a customer commitment, a compliance step. Being faster isn't worth a new failure there.
- Split when the candidate is better at the routine work and worse at the hard cases. Send the routine work to the new model and keep the hard cases where they are. That's more setup, and it's often the best result.
Also decide who signs off. For a workflow that touches customers, it should be the person who owns the outcome, with the comparison in front of them.
Things that break quietly in an upgrade
Most upgrade problems aren't dramatic. They're small changes in behavior that pass a casual look. These are the ones I check for.
Output format. If your workflow expects a specific structure, test that every response still parses. Anthropic's announcement notes that the pre-release version of Sonnet 5.5 given to an outside evaluator had a structured-output bug, since fixed. That's a normal thing to happen around a launch, and it's exactly the kind of problem a saved test set catches on day one.
Settings that no longer mean the same thing. Anthropic's migration notes say that running Sonnet 5.5 with thinking turned off requires a new setting first. If a developer set your workflow up months ago, ask them to read the migration notes before changing the model name.
More effort isn't always better. Anthropic reports that Sonnet 5.5 scored lower at its Max effort setting than at the next level down on one coding benchmark, because it took on extra work that ran out of time or edited things outside the task. Test the setting you'll actually run. Don't assume the highest one is the safest.
Length and tone. A new model may write shorter replies, longer ones, or more formal ones. If the output goes to customers, read ten of them side by side.
Refusals. Newer models ship with updated safeguards. Anthropic notes that Sonnet 5.5's biology safeguards may flag some legitimate microbiology and virology requests. If you work in a field near a sensitive topic, such as pest control chemicals, medical offices, or security, include those requests in your set.
Caching. If your workflow uses prompt caching to save money, a model change starts the cache from empty, and the cache prices changed too. My post on prompt caching in production covers how to measure that.
Pin the version and keep a way back
Two habits make every future upgrade cheaper.
Name the exact model. Use the specific model name, such as claude-sonnet-5-5, and avoid an alias that moves when the vendor ships something new. With an alias, the upgrade happens to you on the vendor's schedule, and the first you hear of it is a customer complaint.
Make the model a setting. The model name should live in one configuration value, not scattered through the workflow. Then switching is one change, and switching back is the same change in reverse.
When you do switch, don't move everything at once if the workflow is busy. Send a small share of live work to the new model for a few days, or run it in the background on real requests without letting it act, and compare. Keep the old model available until you've seen a normal week go by.
Vendors retire old models on a published schedule, so "never upgrade" isn't an option. A saved test set means the day a model is retired is a routine afternoon and not an emergency.
When you don't need to do any of this
Not every use of AI needs a test set. If you use a chat app to draft a post or think through a decision, and you read the result before anything happens, just use the new model. The apps usually move you to it automatically.
The test matters when a model is wired into a process that runs without someone reading each result: sorting, extracting, routing, replying. The less a person looks at each output, the more the saved cases are doing the looking for you.
What I'd do this week
If you have a workflow running on Sonnet 5 or GPT-6 Sol today:
- Find out which exact model name it uses and where that's set.
- Save 20 to 30 real cases with the result you'd accept.
- Run both models and write down the comparison.
- Decide by the rules above, and keep the old model one setting away.
The whole exercise is a few hours the first time. After that, each new release is a quick re-run and a decision you can explain.
This is part of what I do when I build or look after workflow automation and integrations: the test cases come with the workflow, so the next model release is a small job. I'm in Southwest Florida and work remotely across the US. If you have a process running on a model you've never re-checked, tell me what it does.
For where these workflows usually start, see business automation with AI.