The Millisecond Budget: Engineering Faster AI Experiences

Teams often respond to a slow AI feature by choosing a faster model. Sometimes that helps. Often the model is only one section of a much longer path.

Measure the Whole Journey

Break one user request into observable segments: connection setup, authentication, input processing, retrieval, model queue, first token, tool calls, final generation, safety checks, and rendering. Track typical and slow-path latency separately. An average can look healthy while a small group of users repeatedly waits far too long.

The most useful measures are experiential:

• Time until the interface acknowledges the request.
• Time until the first meaningful information appears.
• Time until the user can make the next decision.
• Time until side effects are confirmed.

Spend Time Where It Buys Quality

Not every step deserves the same speed. A search suggestion should feel immediate. A consequential analysis may justify additional reasoning and verification. Route simple requests to a smaller model, retrieve in parallel where safe, cache stable context, and avoid sending a lifetime of history into every turn.

Tool calls need their own budgets. Set timeouts, expose progress, and provide a useful fallback when a dependency is slow. Streaming can improve perceived speed, but it does not excuse an uncertain answer that changes direction halfway through.

Optimize the Slow Path

Test cold starts, long documents, multilingual input, poor networks, and tool failure. The polished demo path is rarely the experience that damages trust.

Latency engineering is product design expressed in time. A good system feels attentive immediately, spends extra seconds only when the result benefits, and never leaves the user wondering whether anything is happening.