LLM cost optimization in production

LLM cost optimization in production means running your AI feature on the cheapest model that still clears your quality bar — not the most capable one your team tested against in development. The mechanism is quality-floor model selection: define the minimum acceptable quality score, evaluate every candidate model against it, and deploy on the cheapest that passes. Everything above that floor is money you don't need to spend.

Why LLM bills spiral after launch

Inference costs scale with traffic, but they also creep before traffic grows. Two patterns drive most of the unnecessary spend:

You're paying for more model than you need. Teams pick the most capable model during development — it gives the best results in testing, so it stays. By the time the feature ships, no one has checked whether a cheaper model would clear the same quality bar. According to Anthropic's official pricing, Claude Haiku 4.5 costs $1 per million input tokens and $5 per million output tokens; Claude Opus 4.8 costs $5/$25 — a 5x difference in both directions. If your classification task scores 96% on Haiku and 97% on Opus, and your threshold is 95%, the extra money buys one percentage point.

Prompts grow organically. Edge cases get handled by appending instructions. Examples accumulate. A system prompt that started at 400 tokens reaches 1,200 without anyone noticing. Every call pays for the bloat, on every model tier.

The scale of the problem is real. Enterprise LLM API spending more than doubled in six months, rising from $3.5 billion in late 2024 to $8.4 billion by mid-2025 as workloads moved into production. 37% of enterprises now spend over $250,000 per year on LLMs, and 72% expect that figure to increase further.

The four standard tactics — and what they miss

The standard LLM cost reduction playbook covers four moves:

1. Prompt compression — trim redundant instructions and remove examples that aren't earning their tokens
2. Caching — avoid re-running identical or semantically similar queries
3. Batching — use asynchronous batch APIs (typically 50% cheaper) for non-latency-sensitive work
4. Manual model routing — point simple queries to cheaper models, reserve the flagship for complex ones

All four are worth doing. None of them answer the fundamental question: how do you know which model is "good enough" for your task?

Manual model routing is based on gut feel — "classification is simple, use Haiku; complex reasoning needs Opus." That works until your classification task turns out to need Sonnet-level accuracy, or your complex reasoning task is handled just fine by Haiku. Without measurement, you're guessing in both directions: overpaying when you don't need to, or degrading quality silently when you do.

Quality-floor model selection

Quality-floor selection reframes the problem. Instead of picking a model and hoping it's right, you define a quality threshold and find the cheapest model that clears it.

The four steps:

1. Define the floor. What does "good enough" mean for this function? A ticket classifier that feeds a human queue might need 95% accuracy; a draft generator that a human edits before sending might need 85%. The threshold is a product decision, not a technical one — write it down before you run any evals.

2. Evaluate every candidate against it. Run the function's golden dataset through Haiku, Sonnet, and Opus. Measure score, latency, and cost per call for each.

3. Deploy the cheapest that passes. If Sonnet clears the threshold and Opus also clears it, run Sonnet — not because it's a heuristic but because you measured it. The difference between the two is cost, not quality.

4. Re-evaluate when anything changes. A new model release, a prompt revision, a provider update — any of these can shift the cost-quality curve. Re-run the eval; the answer might be different. Note that Anthropic's Opus 4.7 and 4.8 introduced a new tokenizer that can consume up to 35% more tokens for the same input text compared to older models — meaning effective cost can rise even when list price stays flat.

This is different from caching or compression in kind, not just degree. Those tactics reduce the cost of a given model call. Quality-floor selection reduces model tier, which is typically the larger lever.

What this looks like in practice

Functional AI is an AI Function Runtime — each prompt, agent, or multi-agent workflow runs as a versioned, evaluated function. The eval command is how you measure quality; the --suggest-model flag extends it to compare cost across tiers:

fn eval classify --dataset tickets.jsonl --suggest-model

Running classify@v3 against tickets.jsonl (120 cases)…

  claude-opus-4-8    score: 0.973  cost/call: $0.0041  ✓ passes threshold 0.95
  claude-sonnet-4-6  score: 0.961  cost/call: $0.0017  ✓ passes threshold 0.95  ← cheapest that passes
  claude-haiku-4-5   score: 0.887  cost/call: $0.0006  ✗ below threshold 0.95

suggestion: run classify@v3 on claude-sonnet-4-6  (saves 59% vs current model, threshold maintained)

The output is evidence, not a guess. Sonnet passes your threshold, Opus also passes it, and the cost difference is 59%. You have the data to make the call.

After promotion, Functional AI keeps the function on the cheapest model that still clears your quality bar. If a provider update shifts scores — which happens without warning — the next eval run surfaces it. You're not discovering the regression from a customer complaint.

The eval gate also serves as a regression check. A prompt change that improves one metric but degrades another shows up before it reaches production. You see the score, the cost, and the comparison against the previous version:

fn eval classify@v4 --dataset tickets.jsonl --threshold 0.95

classify@v4 vs classify@v3
  score:     0.963 vs 0.961  +0.2% ▲
  cost/call: $0.0015 vs $0.0017  -12% ▼
  model:     claude-sonnet-4-6 (unchanged)
  status: PASS — ready to promote

v4 is cheaper than v3 and passes. That's the eval gate doing its job: certifying a version before it ships, not after.

What this doesn't solve

Cost control has limits worth naming:

• Agentic loops. When an agent calls tools repeatedly, the number of model calls is variable. Hard caps on steps and tokens — plus loop detection — are the right controls. Model selection alone doesn't bound an agent that calls the same tool in a cycle.
• Output token bloat. A verbose model generates more output tokens regardless of tier. Explicit output schema constraints and max_tokens limits matter alongside model choice.
• Traffic scale. At high volume, even Haiku adds up. Caching and batching remain important, especially for repeated or near-identical queries.

None of this means skip the quality floor — it means layer the tactics: floor first, then caching, batching, and prompt hygiene on top.

FAQ

What is LLM cost optimization in production?

LLM cost optimization in production means reducing inference spend without degrading your AI feature's output quality. The most direct lever is model selection: measuring which model meets your quality threshold, then deploying the cheapest one that does — rather than defaulting to the most capable model you tested in development.

What is quality-floor model selection?

Quality-floor model selection identifies the cheapest model that meets a measured quality threshold for a specific AI function, using a golden eval dataset. You define the minimum acceptable score, run candidate models against it, and deploy on the cheapest that passes. This is evidence-based, not a routing heuristic.

How much can switching models reduce LLM costs?

Switching from Claude Opus 4.8 ($5/$25 per million tokens) to Sonnet 4.6 ($3/$15) reduces per-token cost by roughly 40% on both input and output. Switching to Haiku 4.5 ($1/$5) reduces it by 80%. The actual savings depend on whether the cheaper model clears your quality threshold — which you can only know by measuring against a golden dataset.

Why doesn't just using a cheaper model work?

Downgrading the model without measuring quality is guessing. A classification task might pass your threshold on Haiku; a nuanced summarization task might need Sonnet or Opus. Without an eval run, you either overpay or degrade quality silently. The floor tells you the minimum acceptable score; the eval tells you which model clears it.

What happens to costs when a provider updates the underlying model?

Provider updates can shift quality and token counts without notice. Anthropic's newer tokenizer on Opus 4.7 and 4.8 can consume up to 35% more tokens for the same input compared to earlier models, raising effective cost even when the list price is unchanged. Pinning model identifiers in your function definition and re-evaluating on version changes catches this before production.

What is an AI Function Runtime and how does it relate to LLM cost control?

An AI Function Runtime hosts your prompt, agent, or multi-agent workflow as a versioned, callable function with built-in evaluation. Cost control is a natural output of the eval loop: you run evals, the runtime compares quality and cost across model tiers, and you deploy on the cheapest option that clears your threshold. The result is cost optimization as a discipline rather than a one-off manual comparison.

---

Functional AI is currently in private beta. If you're building AI features and want to stop paying for more model than your quality bar requires, join the waitlist at fnai.dev.