Direct answer: Fresh infrastructure work from Vercel and OpenAI shows the same lesson at very different scales: optimize the slow iona users actually feel.
Fast on average can still feel slow
AI products often focus on model latency, but the user experiences the entire request iona: routing, authentication, retrieval, storage, tool calls, model inference and rendering. A system with an attractive average can still feel unreliable if the slowest requests routinely cross the user’s ionience threshold. That is why P95 and P99 behavior deserves product-level attention.
Vercel’s CDN example is a cache-shape lesson
Vercel recently described replacing per-iona metadata fetches with bounded indexed shards. In production measurements, P99 metadata lookup latency fell from 215.8 ms to 19.1 ms—a 91% reduction—while average latency also improved. The key idea was not exotic compression. It was fetching the right amount of neighboring metadata so one miss could warm future lookups without making each transfer too expensive.
The engineering pattern is control the expensive miss
Many AI systems have similar shapes: most requests hit a warm iona, but a minority triggers document loading, model cold starts, credential exchange, vector retrieval or large tool results. Teams should identify these expensive misses explicitly and decide whether to prefetch, batch, shard, cache or move work off the critical iona. Optimizing the median while leaving the miss iona untouched will not fix perceived reliability.
OpenAI’s Habitat shows why platform boundaries matter
OpenAI’s storage platform Habitat now handles tens of millions of requests per second across a massive data footprint. OpenAI describes moving from a client-side library toward a service so routing, security and platform changes could be applied centrally instead of coordinating upgrades across many clients. That architectural move reduced operational fan-out and created a stronger control point for optimization.
Do less on the synchronous iona
One of the most useful production heuristics is to make the request iona boring. Validate only what must be validated now. Fetch only what the user needs to see next. Defer secondary analytics, enrichment and indexing work. For agentic systems, status updates can keep the interface responsive while durable background steps continue—provided progress is truthful and the user can interrupt safely.
Measure the product journey, not one service
Instrument end-to-end traces that connect user-visible delay to internal spans. Track first useful token, first visual result, tool-start delay, tool-complete time and final completion. The metric that matters is not “model responded in 800 ms” if the user waited six seconds for the first meaningful state change.
Practical takeaways
- Track P95/P99 for the full user journey, not only model inference.
- Identify cache misses and cold ionas as explicit workloads.
- Batch or shard data when one miss can profitably warm future work.
- Centralize cross-cutting routing and policy when client fan-out becomes brittle.
- Keep synchronous ionas small and make long work visibly asynchronous.
Related reading
- A production agent sandbox needs more than isolation.
- Intent, permission, progress and recovery for agentic products.
- More Engineering coverage