LLM apps fail in production for boring reasons: a token bill nobody modeled and a latency budget nobody set. Both are controllable.
A prototype that costs a few dollars a day can become a feature that costs thousands a month the moment real traffic arrives. The good news is that LLM cost and latency are engineering problems with well-understood levers — but only if you instrument them from day one instead of discovering the bill at the end of the month.
Build a cost model before you build the feature
Cost is roughly requests × tokens-per-request × price-per-token. Estimate each before you write the feature: how many calls per user action, how big is the context you're stuffing in, and which model tier are you using. Most surprise bills come from oversized context — retrieving twenty chunks when three would do, or replaying entire conversation histories on every turn.
The levers that actually move the bill
- Right-size context: retrieve and rerank to the few chunks that matter instead of padding the prompt.
- Cache aggressively: identical or near-identical requests shouldn't pay twice; prompt caching cuts the cost of stable system prompts.
- Route by difficulty: send easy cases to a small, cheap model and reserve the frontier model for the hard ones.
- Trim output: cap max tokens and ask for structured, concise responses.
Token spend should never be a surprise. If you can't see it per feature, you can't control it.
Latency is a budget, not an afterthought
Users feel latency long before they see your bill. Set a budget per interaction and design to it: stream tokens so the response feels instant, parallelize independent calls, keep retrieval fast, and cache what you can. A model that's 200ms faster and 30% cheaper but slightly less capable is often the right production choice once you've measured what 'capable enough' means on your evals.
Instrument everything
Log tokens, cost, latency, and cache hit-rate per request and per feature. Once spend is visible, optimization becomes obvious — you'll see exactly which call to cache, which prompt to compress, and which path to route to a cheaper model. The teams that control LLM cost aren't the ones with the cleverest tricks; they're the ones who can see what they're spending.
