Timeouts, retries, and the agent that took down staging
By Pavan Jegurupati, Founder, InferLearn
A participant in the last System Design cohort pointed an evaluation agent at a staging environment on a Thursday afternoon. By the time anyone looked at the dashboard, a single search endpoint was taking a thousand requests a minute from one laptop. Nothing was under attack. Every layer was behaving exactly as it had been configured to.
The incident
The agent called a retrieval tool. The tool called an internal search service. The search service called a vector store that had just been redeployed and was cold.
What we saw
Latency on the search service went from 90 ms to about four seconds while the new pods warmed their caches. That is slow, but it is not an outage - left alone, it would have settled within two minutes.
It did not get two minutes. The tool wrapper had a three-second timeout and three retries. The agent framework wrapped that in its own retry. And the agent itself, seeing a tool error come back, decided the sensible next step was to call the tool again with a slightly reworded query.
Three layers of retry multiply. They do not add.
What we changed
We stopped retrying in the middle. Retries belong at exactly one layer - the one that knows what the whole operation costs and whether it is still worth finishing. Everywhere else, a failure should propagate.
Timeouts, retries & the "just one more" reflex
The reflex is understandable. A retry is the cheapest-looking line of code in any codebase: one wrapper, and a whole class of transient failure disappears from your error rate. The cost is invisible until the day the dependency is slow rather than down.
What we saw
A timeout that is shorter than the dependency's real p99 is not a safety mechanism. It is a load generator. Ours was three seconds against a service whose p99 under a cold cache was four - so every single call timed out, and every single call retried, and each retry arrived while the previous one was still executing on the server.
The server never got to finish anything. That is the part people miss: a client-side timeout does not cancel the work. Unless the connection is closed and the server checks for it, the original request is still running when its replacement arrives.
What we changed
We set the timeout from the measured p99 plus headroom, not from a round number that felt responsive. And we made cancellation real - the tool now passes a deadline the server actually checks.
# Deadline travels with the call, so the server can abandon work the
# client has already given up on.
async def search(query: str, budget: RetryBudget) -> list[Hit]:
## NOTE: this line is inside a fence - it must never reach the contents rail
deadline = budget.deadline_for(p99_seconds=4.0, headroom=1.5)
while budget.can_attempt():
try:
return await client.search(query, deadline=deadline)
except Transient as err:
budget.record_failure(err)
await asyncio.sleep(budget.next_backoff())
raise Exhausted(budget.summary())Why max_retries is the wrong knob
max_retries describes one call in isolation. The thing that hurts you is the
aggregate: how much extra load the whole system is willing to manufacture while
a dependency is unhealthy.
| Strategy | Blast radius under a slow dependency | When it is right |
|---|---|---|
| Fixed count, no backoff | Multiplies load by the retry count, immediately | Almost never |
| Exponential backoff | Bounded, but every client still retries | A single client, low concurrency |
| Backoff with jitter | Spreads the herd across the window | Many independent clients |
| Retry budget | Caps retries as a share of total traffic | Anything with a fan-out |
| Circuit breaker | Drops to zero, then probes | A dependency that fails hard, not slowly |
Budgets, not counts
A retry budget says: across this client, retries may be at most ten percent of successful requests. When the dependency is healthy, ten percent of a large number is plenty. When it is unhealthy, successes collapse - and so does the retry allowance, automatically. The system stops amplifying at precisely the moment amplification would hurt.
A note on jitter
Full jitter beats equal jitter in almost every simulation worth running, and both comprehensively beat none. Without jitter, a backoff schedule does not break up a thundering herd - it synchronises one.
Before you add a retry
The checklist we now run in the cohort, in order:
- Is the failure actually transient? A 400 will fail identically forever.
- Is the operation idempotent, or does a retry risk double-charging someone?
- Is there already a retry at a lower layer? Find it before you add another.
- What is the dependency's measured p99 - not its target, its measurement?
- Does the timeout sit above that p99, with headroom?
- Does cancellation propagate, so the server can abandon abandoned work?
- Is there backoff, and does it have jitter?
- Is there a ceiling on total retries as a share of traffic, not per call?
- What does the caller do when the budget is exhausted - degrade, or fail?
- Will the dashboard show retries separately from first attempts?
The last one catches more incidents than the other nine combined. If retries are folded into your request count, the graph of a retry storm looks exactly like the graph of a traffic spike, and you will spend the first twenty minutes of the incident scaling up the thing that is already drowning.
Agents make all of this sharper, because an agent is a retry loop that can rewrite its own input. Give it a tool that fails slowly and it will keep trying variations of the same doomed call for as long as you let it. The budget is what stops it - not the prompt.
Keep reading
