Direct answer: Fresh infrastructure work from Vercel and OpenAI shows the same lesson at very different scales: optimize the slow paths users actually feel.
Fast on average can still feel slow
AI products often focus on model latency, but the user experiences the entire request path: routing, authentication, retrieval, storage, tool calls, model inference and rendering. A system with an attractive average can still feel unreliable if the slowest requests routinely cross the user’s patience threshold. That is why P95 and P99 behavior deserves product-level attention.
Vercel’s CDN example is a cache-shape lesson
Vercel recently described replacing per-request metadata fetches with bounded indexed shards. In production measurements, P99 metadata lookup latency fell from 215.8 ms to 19.1 ms—a 91% reduction—while average latency also improved. The key idea was not exotic compression. It was fetching the right amount of neighboring metadata so one miss could warm future lookups without making each transfer too expensive.
The engineering pattern is control the expensive miss
Many AI systems have similar shapes: most requests hit a warm path, but a minority triggers document loading, model cold starts, credential exchange, vector retrieval or large tool results. Teams should identify these expensive misses explicitly and decide whether to prefetch, batch, shard, cache or move work off the critical path. Optimizing the median while leaving the miss path untouched will not fix perceived reliability.
OpenAI’s Habitat shows why platform boundaries matter
OpenAI’s storage platform Habitat now handles tens of millions of requests per second across a massive data footprint. OpenAI describes moving from a client-side library toward a service so routing, security and platform changes could be applied centrally instead of coordinating upgrades across many clients. That architectural move reduced operational fan-out and created a stronger control point for optimization.
Do less on the synchronous path
One of the most useful production heuristics is to make the request path boring. Validate only what must be validated now. Fetch only what the user needs to see next. Defer secondary analytics, enrichment and indexing work. For agentic systems, status updates can keep the interface responsive while durable background steps continue—provided progress is truthful and the user can interrupt safely.
Measure the product journey, not one service
Instrument end-to-end traces that connect user-visible delay to internal spans. Track first useful token, first visual result, tool-start delay, tool-complete time and final completion. The metric that matters is not “model responded in 800 ms” if the user waited six seconds for the first meaningful state change.
Design for p95, not the demo
Teams often optimize AI experiences around a fast median response because that is what a happy-path demo shows. Users, however, remember the outliers. A tool call that occasionally waits ten seconds, a retrieval step that stalls or a model response that streams slowly can make the whole product feel unreliable even when average latency is excellent. Engineering teams should therefore track p50, p95 and p99 across the entire interaction chain, not just the model API. Each external dependency needs its own budget and timeout.
The interface should be built around those distributions too. Progressive results, cancellable work, visible sub-steps and sensible fallback states make long-tail events survivable. If the agent can continue useful work while one dependency is slow, expose that progress instead of freezing the screen. If a slow tool is optional, skip it and explain the degraded result. Caching stable context and precomputing expensive metadata can remove latency before the model is even called. The broader point is that speed is not one metric; it is a reliability contract. Products feel fast when users know what is happening and can keep control during the slowest realistic cases, not only when a benchmark reports a low average.
Practical takeaways
- Track P95/P99 for the full user journey, not only model inference.
- Identify cache misses and cold paths as explicit workloads.
- Batch or shard data when one miss can profitably warm future work.
- Centralize cross-cutting routing and policy when client fan-out becomes brittle.
- Keep synchronous paths small and make long work visibly asynchronous.
Related reading
- A production agent sandbox needs more than isolation.
- Intent, permission, progress and recovery for agentic products.
- More Engineering coverage