Search latency
Cut median search latency by 34% with caching, and reduced CPU by 31% on search and 71% on its category API.
- −34%
- Median search latency
- −78%
- Calls to the category API
- −31%
- Search API CPU at the same traffic
01
What happened
- The category API behind search got slow in production, and search, which calls it on every request, slowed down with it.
- Cause: a tool polling a status endpoint that ran heavy calculations, overloading the category API.
02
What I changed
- A short-lived cache on the status endpoint the tool was calling.
- Rate limits on the category API that endpoint was using downstream.
- A smart caching strategy in the search API to cut calls to the category API.
03
Outcome
- Calls from search to the category API dropped 78%, from about 280 to 60 per second, at the same search traffic.
- Category API calls per search request: from about 3.6 to 0.9.
- Search latency: −34% at p50, −19% at p95 and −19% at p99.
- Search API CPU down 31% at the same traffic, about 30% less per request.
- Category API CPU down 71%, entirely from fewer calls.