How We Cut LLM Costs 40–70% Without Losing Quality
Every agent we inherit is spending more than it needs to, and the fix is rarely "use a cheaper model". This is the order we work through — free wins first, quality trade-offs last, everything measured against the eval suite so nothing gets worse.
- Published
Step 1 — measure before touching anything
- Pull a month of usage: tokens in, tokens out, cached vs uncached, per route or task type. If the app logs per-response usage, use that; otherwise the provider’s usage reports.
- Compute cost per completed task, not per request. A cheaper request that needs three retries is not cheaper.
- Confirm an eval suite exists. If not, build one first — cost work without evals is how quality quietly dies.
Steps 2–5 — the free wins
| Step | Lever | Typical saving | Quality risk |
|---|---|---|---|
| 2 | Prompt caching: stable prefix first, volatile content last, no timestamps up top | 30 – 60% of input cost | None |
| 3 | Input hygiene: trim tool results, clear stale context, stop resending whole documents | 10 – 30% | None if evals hold |
| 4 | Loop hygiene: step caps, parallel tool calls, return errors instead of retrying blindly | 10 – 25% | None |
| 5 | Batch the non-urgent: nightly jobs through the batch API | 50% on that traffic | None |
| 6 | Effort tuning per route: low for classification, high only for reasoning-heavy steps | 15 – 40% of output cost | Low — measure |
| 7 | Model routing: small model for extraction and classification, frontier model for judgement | 20 – 50% | Medium — measure |
- Caching. Render order is tools → system → messages; anything that changes per request must sit after the cache breakpoint. Verify with the cache-read token count — zero means something is invalidating the prefix.
- Input hygiene. Most agents resend far more context than the model needs. Clear old tool results, summarise long histories, send the paragraph, not the document.
- Loop hygiene. Cap steps, budget tokens per task, run independent tool calls in parallel and return all results in one message.
- Batch. Anything that can wait an hour goes through the batch endpoint at half price.
Steps 6–7 — the trade-offs, measured
- Effort. Most routes do not need maximum reasoning. Sweep effort levels per route against the eval suite and keep the lowest that holds the bar — on current models a lower effort setting often matches the previous generation at full effort.
- Routing. Send extraction, classification and summarisation to a smaller model; keep the frontier model for planning and judgement. Measure the split on the suite; keep one model per route so caches stay warm.
Across the agents we have done this on, the free steps alone land 30–50% and the full sequence 40–70%, with the eval pass rate unchanged or better. It takes one to two weeks. We do it as a fixed-price engagement or teach it in a half-day session.
Frequently asked questions
Should we just switch to a cheaper model?
Last, not first. Caching and input hygiene usually save more with zero quality risk. Switch models only with an eval suite that measures the change.
Does caching really work for agents?
Very well, if the prefix is stable. Long system prompts and tool lists are exactly what caching is for. The common bug is a timestamp or request ID at the top invalidating everything.
How fast do savings show up?
Caching and hygiene changes show in the next day’s usage. Effort and routing take a week of measuring to do safely.
