Timeouts and retries

Explanation

Wait for a response

A caller asks a remote service for CI status. Network travel and processing both take time; the caller waits for the response.

A successful response supplies evidence about the request. Every time in this sequence is an illustrative policy value.

0s The caller asks for CI status. The remote service must answer.
7s The service processes the request. The caller is waiting.
14s A response arrives in time. The caller can use the reported status.
21s Some responses take too long. A timeout bounds one attempt.

A timeout is not a rollback

The caller stops waiting when its attempt timeout expires. Work at the remote service may continue.

Missing a response means the outcome is unknown. Cancellation and rollback require their own implementation.

0s The caller is still waiting. The remote request is in progress.
7s The attempt timeout expires. The caller stops waiting now.
14s Remote work may still continue. A timeout does not undo that work.
21s Report the outcome as unknown. Do not invent a failed build result.

Retry only when safe

A retry is another attempt at the same operation. A status read can usually be retried, but effectful operations need a safe retry contract.

Decide retry safety from the operation contract. An idempotency key can help protect a write when the service implements it.

0s The first status read timed out. Another attempt may obtain the result.
7s Check whether repetition is safe. This operation only reads CI status.
14s Send a second bounded attempt. It is the same question, asked again.
21s Use the response if it arrives. A write would need its own contract.

Back off before retrying

After a shared failure, callers can retry together and add another burst of load. Backoff waits; jitter varies the wait between callers. These delays are illustrative.

Different retry delays spread arrivals. They do not guarantee recovery, and total attempts still need a bound.

0s Three callers encounter the same failure. Immediate retries would form a new burst.
7s Each caller chooses a retry delay. Jitter makes those delays different.
14s Watch the retries arrive at different times. The service still may not recover.
21s Each caller has made two attempts. Keep further retries within a limit.

Bound the whole operation

An attempt timeout and an overall deadline solve different problems. In this illustrative policy, two attempts and their delay must fit within twelve seconds. Animation time is slowed for explanation.

Five seconds for the first attempt plus a two-second delay leaves at most five seconds for a retry, less overhead. The deadline gate rejects further attempts when the shared budget reaches zero.

0s Example policy: a 12-second deadline. The gate permits the first attempt.
7s Attempt 1 times out after 5 seconds. The shared budget has 7 seconds left.
14s A 2-second backoff leaves at most 5s. The second attempt uses that remainder.
21s The deadline expires: no third attempt. Return unknown and both timed-out tries.

Read the guide

Helpful first: Tool design.

The idea

A timeout limits how long a caller waits for an attempt. An overall deadline limits the whole operation, including retry delays. Neither proves that remote work was cancelled or rolled back. gRPC explicitly makes the server application responsible for stopping work it spawned. Treat a missing response as missing evidence, not evidence that nothing happened.

Where it helps

An engineer can notice a stuck request and stop clicking. An unattended agent needs that stopping rule in executable tool code. Retries can recover transient failures, but they also consume capacity; independent retry loops at multiple layers multiply downstream attempts. Review the runtime, client library, and service policy together.

Teams delegating remote tool use need a reliable stopping rule before leaving agents unattended. This chapter makes existing reliability knowledge actionable through a small runtime policy and a fault-injection exercise.

A release assistant reads CI status before drafting a release summary. In an illustrative policy, the runtime allows two total attempts, up to five seconds each, within a twelve-second tool budget. The first times out; a sampled two-second backoff leaves at most five seconds for the retry, less any runtime overhead. If that also expires, return status_unknown with the attempt history. A tool timeout must not become a claim that the build failed.

What you will be able to check

Distinguish an attempt timeout from a shared operation deadline in a trace.
Implement one bounded retry policy for a read-only tool and demonstrate its stop condition.
Explain why a timed-out write needs reconciliation or documented idempotency before retrying.

What changes in the team

Today: An engineer refreshes CI, checks an outage dashboard, and decides whether to try again.
With an agent: An agent calls a typed status tool; runtime code owns the timer, retry classification, attempt counter, and terminal result.
Still your responsibility: Engineers choose safe operations and limits, inspect failure traces, and decide how to handle unresolved outcomes.

Work through an example

Define the operation. Read CI status for a specific build ID. Keep that identity and the budget across repeated agent requests belonging to the same operation. Classify the endpoint by its documented semantics: RFC 9110 cautions against automatically retrying non-idempotent requests unless safety or non-application is known. Define an operation as this task’s bounded read, not every future read of the build forever. A later scheduled poll or explicit new task can receive a new budget; a model rephrasing the exhausted request cannot grant one itself.
Set the deadline once using a monotonic clock. Before each attempt, compute remaining time; cap the attempt timeout by that remainder. Include connection setup, response reading, and sleeps in the budget. Check what your client’s timeout actually covers, especially streaming responses and connection establishment.
Retry only allowed transient failures. Sleep with a capped backoff and jitter, meaning randomized delay; identical delays can synchronize clients again. Check remaining time after sleeping and before sending. Honour server retry guidance when applicable; stop if waiting would exhaust the budget. A deadline stops caller waiting; propagate cancellation where supported and make spawned work cooperate, without treating that signal as proof of rollback.
Finish with evidence. On success, return the observed status and build ID. On exhaustion, preserve the unknown outcome, deadline, and attempts in the trace. For a write, use the provider’s documented idempotency contract or reconcile the effect first. Stripe API v1, for example, can replay saved errors and permits pruning keys after at least 24 hours; a key is not a universal, permanent duplicate shield.

The engineering decision

Choose limits from measured dependency latency and the task’s tolerance for waiting. Identify one owner for application retries and inspect lower-layer behaviour: gRPC can transparently retry some failures even without an explicit retry policy. Count actual downstream calls, not only model-visible tool calls. These example numbers are teaching values, not production defaults.

Resetting the budget when the model rephrases the same tool request bypasses the intended limit.
Treating every error as transient retries invalid arguments or unchanged credentials without fixing the cause.
Assuming a timeout stopped server work can cause duplicate writes when a retry follows a lost acknowledgement.

Try it with your team

Make a slow CI read stop predictably

Stub a read-only CI endpoint with success, delayed response, retryable persistent failure, and invalid-argument modes; log each arrival and completion. Disable hidden client retries for the two-call test, then separately inspect the real client’s retry settings.
Implement the illustrative two-attempt, twelve-second policy. Inject a clock and a fixed two-second delay for a deterministic test.
Run all four modes and record operation ID, attempt number, remaining budget, and terminal result.
Ask for the same logical read after exhaustion; verify the runtime rejects it without another downstream call.
Let a delayed stub finish after the caller times out; verify the result stays unknown at the caller, while the completion log proves remote work continued. Verify an invalid argument is rejected without a retry.

You have an answer when…

Success returns a real stub status; persistent failure returns an explicit unknown outcome within the budget plus a measured scheduling tolerance.
The failure trace shows at most two application attempts, one shared deadline, and no downstream call after closure.
A non-retryable error causes one downstream call, and a late remote completion does not become an accepted result for the expired attempt.

Read further

Can a model request the same logical operation again and accidentally receive a fresh deadline or attempt allowance?

Original video comparisons