Retrieval Latency: Where the Milliseconds Hide

"Slow retrieval" is not a diagnosis

When someone says retrieval is slow, they are describing a symptom that spans four different systems, each with its own failure mode. The request parses a query, turns it into a vector, searches a vector store, then re-ranks and filters the results. Any one of those four can be the bottleneck, and they trade off against each other, so tuning the one you guessed at often moves latency somewhere else. Every additional fifty milliseconds shows up as lost engagement and abandoned sessions, and it compounds, because retrieval usually sits in front of a generation call that the user is already waiting on.

The only way out is measurement. You cannot fix a total you have not decomposed, and you cannot decompose a total you have not traced. The discipline is to assign a latency budget to each of the four phases before you optimize anything, then instrument the request path so every phase reports its own timing and its own outliers. Once you can see that embedding is eating thirty milliseconds and retrieval is eating forty, the argument about what to optimize stops being an opinion and becomes arithmetic. The milliseconds are hiding in specific places, and the whole job is to make those places visible.

Budget the four phases, then trace them

Start with an explicit per-phase budget for a target end-to-end latency, then measure actuals against it across at least a thousand requests so you see the tail, not just the median. The table shows a worked hundred-millisecond budget alongside the lever that pulls time back out of each phase when the actual overshoots.

The budget is a contract, not a prediction. When a phase blows through its allotment you know exactly where to spend effort, and when it comes in under you know you can borrow that headroom for a phase that needs it. Enforce the budget in CI load tests so a regression fails the build instead of surfacing as a user complaint two weeks later.

Trace a real request to see it. A support-search query comes in at 140 ms end to end against a 100 ms budget, and the team assumes the vector store is the problem. The trace says otherwise: parsing 12 ms, embedding 61 ms, search 44 ms, re-ranking 23 ms. Embedding is double its budget because every query makes its own network call. Adding an embedding cache for the top 500 repeated queries turns those into 1 ms lookups and drops the phase to an amortized 22 ms, pulling the request to 101 ms without anyone touching the index. Only then does search, now the largest phase, become worth tuning, and lowering efSearch from 200 to 96 against a recall floor trims it to 31 ms. The total lands at 88 ms, under budget, and every millisecond was reclaimed from a place the trace named rather than a place someone guessed.

Reclaim milliseconds phase by phase

• Enable distributed tracing across every retrieval service so each phase emits its own timing, then benchmark a baseline across at least a thousand requests and read the p95 and p99, not just the average, because the tail is where users churn.
• Attack embedding first if it overshoots: batch queries where you can and cache embeddings for frequently seen queries and documents, since a cache hit turns a thirty-millisecond network call into a microsecond lookup.
• Tune the vector search with intent: lower HNSW efSearch until recall drops to your floor, and preload hot partitions into memory so the common queries never touch disk.
• Stream partial results by returning the top-k as soon as the search completes and finishing re-ranking in the background, so the user sees results while the last twenty milliseconds of work is still running.
• Run latency drills that simulate traffic spikes and failovers, and enforce the per-phase budget in CI load tests so a regression blocks the release rather than reaching production.

Where retrieval latency work goes sideways

• Optimizing an unmeasured hotspot. The team tunes the vector index because it feels slow, when the real cost was a synchronous embedding call. Fix: trace all four phases first and spend effort only where the budget is actually blown.
• One index setting for every workload. A single efSearch value is tuned for the hardest query and taxes every easy one, inflating the search phase across the board. Fix: set efSearch against your recall floor and consider per-collection settings for distinct workloads.
• Serialization tax on every hop. Each service boundary JSON-encodes and decodes the payload, and the hops add up to more than any single phase. Fix: cut unnecessary hops and use a compact wire format between retrieval services.
• Measuring the median and ignoring the tail. The average looks fine while p99 users wait a full second on cold caches. Fix: report p95 and p99, run latency drills, and pre-warm caches so the tail does not spike on the first request after a lull.
• Blocking on full re-ranking. The system re-ranks the entire candidate set before returning anything, so the user waits on work they may never scroll to. Fix: stream the top-k immediately and finish re-ranking the remainder in the background.

Find your hidden milliseconds this week

• Turn on distributed tracing across the retrieval path so every phase reports its own timing and outliers.
• Set a per-phase latency budget, for example 10 ms parsing, 30 ms embedding, 40 ms retrieval, and 20 ms re-ranking, and measure actuals across at least a thousand requests.
• Precompute and cache embeddings for your top queries and documents so repeat lookups skip the network call.
• Tune HNSW efSearch against a recall floor and preload hot partitions into memory to keep common queries off disk.
• Enforce the budget in CI load tests and read p95 and p99, so a latency regression fails the build instead of the user.