LLM Cost Optimization Patterns

Anthropic, OpenAI, and Google all offer prompt caching that reuses the KV cache for repeated prompt prefixes across requests, typically reducing input-token cost by ~10x for the cached portion. This is not a marginal optimization; it is the highest-leverage lever in most workloads.

Workloads that benefit most: RAG (stable system prompt + retrieved context that repeats within a session), agent scaffolds (system prompt + tool schemas repeat across every step), few-shot prompting (example set repeats across every query), long system prompts (organizational voice guide, safety instructions, style rules).

Implementation: structure prompts so the stable content comes first, variable content last. Prompts are typically cached by prefix, so any change in the early portion invalidates the cache. Version your system prompts and tool schemas separately from the variable inputs. Report cached-input-ratio per system in your cost dashboard — anything below 60% on a cache-friendly workload is a red flag.

Rough sizing: enterprise RAG workloads with poor caching pay 3-8x the input-token cost of the same workload with well-designed caching. For agent workloads, the multiplier climbs to 5-10x because the agent scaffold repeats on every step.

The default of "one premium model for everything" is expensive theater. Different subtasks have different capability requirements; matching model tier to task tier saves 3-10x with minimal quality impact.

Routing dimensions: task complexity (simple classification → mid-tier; complex reasoning → reasoning-mode frontier), risk of the output (low-consequence → cheap; irreversible or user-facing → expensive), latency sensitivity (fast response required → mid-tier or small; deep analysis acceptable slow → reasoning model), and cost budget for the tenant (some customers pay for premium routing; others do not).

Implementation approaches: hand-coded rules (simplest; works for well-understood task categories), LLM-based classifier (cheap classifier decides the tier at request time), semantic router (embedding similarity to labelled examples). The middle option is the pragmatic default — a Haiku / Sonnet / Flash / GPT-5-mini classifier can decide with high accuracy at negligible cost.

Anti-pattern: routing decisions that ignore per-task cost variance. A "route by complexity" system that sends 90% of traffic to the premium tier because "the classifier is not perfect" defeats the purpose. Tune the classifier confidence threshold aggressively; err toward the cheaper tier with a fallback if the cheap answer scores poorly.

For workloads with long prompt histories (multi-turn conversations, agent trajectories, long RAG contexts), the input token count grows quickly. Compression drops that count without proportional quality loss.

Techniques: rolling summarization (turn N-K conversations into a compressed summary before appending turn N), semantic chunking of retrieved documents (return only the most relevant passages, not full documents), tool-output truncation (a tool that returns 10K rows should return the 20 most relevant with a "and 9,980 more" note), and structural compression (remove markdown formatting, whitespace, redundant labels when the model does not need them for accuracy).

The trade-off is measurable — compress too aggressively and task-success drops. Instrument both. Compression ratio and task-success rate should move together on your dashboard. Target the compression level that maintains task-success within 2% of the uncompressed baseline.

An inference-time optimization: a small "draft" model generates candidate tokens, the target model verifies them in parallel. Effective throughput increases, latency decreases, and the semantic output is identical to running the target model alone.

Most enterprise teams do not implement speculative decoding directly — it is baked into modern inference stacks (vLLM, TensorRT-LLM, most frontier-provider APIs). Awareness matters because it affects vendor selection: self-hosting on a stack without speculative decoding pays more than self-hosting on one that does; provider APIs that pass through the throughput benefit are cheaper effectively than those that do not.

Practical action: if you self-host open-weights models, ensure your inference stack supports speculative decoding for the model family in use. For hosted models, factor throughput-per-dollar (not just headline price-per-token) into vendor comparisons.

Frontier providers offer batch APIs (Anthropic, OpenAI, Google) that trade guaranteed sub-24-hour completion for ~50% cost reduction. Suitable for asynchronous workloads: nightly evaluations, bulk data enrichment, retrospective analysis, batched classification.

Implementation is straightforward — the API accepts a batch of requests, returns results within the SLA window. The engineering discipline is recognizing which workloads can move to batch. Anything that does not require synchronous response is a candidate: analytics pipelines, offline eval runs, marketing content generation, knowledge-base ingestion.

Rule of thumb: 10-30% of a mature GenAI program's total token spend can move to batch pricing with no material user impact. Teams that do this systematically drop total cost by that same fraction.

When a workload is high-volume, well-scoped, and has a clear evaluation harness, distilling from a frontier model into a smaller model — either an open-weights model you fine-tune or a smaller provider tier — often yields 5-20x cost improvement with acceptable quality.

The recipe: (1) run the frontier model in production for long enough to accumulate a high-quality dataset of (input, output) pairs, ideally filtered by human review or downstream success signal; (2) fine-tune a smaller model on that dataset; (3) evaluate the fine-tune against a held-out portion of the golden set; (4) route the workload to the fine-tune when it clears a quality threshold, with the frontier model as fallback for out-of-distribution inputs.

Common workloads where this pays off: intent classification, entity extraction, structured-output tasks, style-conformant generation. Common workloads where it does not: open-ended reasoning, tasks with high input variance, tasks where the failure mode is subtle enough that a smaller model quietly gets things wrong.

The trade-off is operational complexity — you now own an inference stack (or manage per-provider fine-tune lifecycle) and a retraining cadence. Justify the distillation project only if the projected annual savings comfortably cover the operational overhead.

The seventh pattern is not a technical optimization — it is a measurement discipline. You cannot optimize what you do not measure, and most teams measure cost per API call rather than cost per successful business outcome.

The right dashboard shows, per team and per system: total spend, cost per successful task, cost per active user, cached-input ratio, and month-over-month trend. Cost per successful task is the north-star metric. If it is trending down, you are winning; if it is trending up, something changed — new model version, prompt scaffold regression, degraded caching, distribution shift.

Budgets should be set per team, per system, and per user tier. Alerts fire at 50%, 75%, 90% of budget with escalating notification. Hard cutoffs at 110% (soft) and 150% (hard) prevent runaway incidents from bankrupting the program while allowing for legitimate spikes with human sign-off.