Users forgive a powerful AI feature more easily than a slow, silent one. Most AI latency is not the model's fault alone — it is retrieval, oversized prompts, serial tool calls, and interfaces that show nothing until everything is finished.
Measure where the time actually goes
Log each stage separately: request handling, retrieval, model time to first token, total generation, and tool calls. Teams often discover retrieval or a slow internal API — not the model — dominates the wait.
Track percentiles, not averages. A feature with a fine average and a terrible 95th percentile feels unreliable, because one visit in twenty is the one users remember.
Stream, and show progress honestly
Streaming the response as it generates changes perceived speed more than most optimisations: users start reading in under a second instead of staring at a spinner for ten. Pair it with truthful progress for multi-step agent work — show the steps as they complete, not a fake progress bar.
Spend the context budget wisely
Every retrieved chunk and history message adds tokens, latency, and cost. Retrieve fewer, better passages; summarise long histories; and drop instructions the model no longer needs. Smaller, relevant prompts are faster and usually more accurate.
Cache what is genuinely repeatable: embeddings for unchanged documents, answers to identical frequent questions, and classification results. Do not cache personalised or permission-sensitive answers across users.
- Limit retrieved chunks; re-rank instead of sending everything
- Run independent tool calls in parallel, not in sequence
- Use a smaller, faster model for routing and classification steps
- Set timeouts and partial-result fallbacks per stage
Keep the rest of the product fast
Isolate AI calls behind their own endpoints and queues so a slow model never blocks normal pages. For background jobs — reports, bulk processing — acknowledge immediately and notify when done instead of holding a request open.
Common questions
For interactive chat, aim for first visible output in about a second and design the UI around streaming. For agent jobs, honest progress matters more than a hard number.
Generally larger models generate more slowly and cost more, but routing simple steps to smaller models and reserving the large model for the hard step often improves both speed and quality.
Related services
Working on something like this?
These are the services behind this guide. Tell us about your project and we will reply with a useful next step — not a sales script.
