An LLM Retry Budget Prevents a Cheap Request Becoming an Expensive Loop

A retry policy needs a limit and a reason. Treat every repeated model call as additional work that must justify its cost.

Classify the failure

Distinguish a temporary transport failure from invalid output and a request that cannot satisfy your policy. A network retry may make sense; repeating the same impossible instruction without changing anything usually provides little information. Record the failure category so a single retry counter does not hide different causes.

Set explicit bounds

Choose a maximum attempt count, elapsed-time limit and cost ceiling for the workflow. Count fallback-provider calls inside the same budget. Give the caller a clear terminal result when the budget is exhausted. A fallback that restarts the retry counter can turn a bounded policy into an unbounded chain.

Protect side effects

Separate text generation from actions such as sending a message or charging an account. Confirm whether an action completed before retrying it, and use the application’s idempotency mechanism where supported. A second model answer does not establish whether an earlier external action succeeded.

Review the exception log

Group exhausted tasks by cause and improve the largest category. Better validation instructions may help formatting failures; service health handling may help transport failures. Recalculate cost per accepted task after the change and retain the original baseline.

Source and scope

Reference checked on 7 September 2026: Gemini API pricing. This article provides an editorial workflow. It does not report a paid account test or guarantee a platform outcome.

Give the workflow one budget owner

Place the attempt count and elapsed-time accounting at the level that can see the entire task. If the model client, tool wrapper and fallback router each retry independently, their limits can multiply. A task-level budget should include every model attempt and any recovery action you choose to count. Pass the remaining budget to lower layers rather than letting them start fresh. Keep the implementation simple enough to inspect in logs. The important property is that one task cannot quietly gain a new allowance every time it changes provider or component.

Use a decision table for failure classes

List the failures your application can actually observe: transport interruption, rate-limit response, schema failure, unsupported request and uncertain external side effect. For each class, define whether to wait, retry, change the input, return a terminal result or ask for human review. Consult the provider’s current error guidance before deciding how its responses should be handled. A retry policy should not be a universal instruction to repeat every unsuccessful request. Some failures require correcting the request, while others require preserving uncertainty about what already happened.

A small example of bounded recovery

Imagine an extraction task with at most three model attempts and a fixed end-to-end deadline. The first output fails schema validation. The second receives a concise correction identifying the invalid field. If the third still fails, the workflow returns an explicit unresolved result with the relevant diagnostic. It does not switch providers and receive three more attempts unless that possibility was included in the original budget. This is an illustrative policy, not a recommended universal limit. Choose the numbers according to the application’s cost and response requirements.

Keep retry instructions connected to the error

For a format failure, provide the specific validation problem rather than appending a long generic warning. For an unsupported field value, state the allowed values or permit an explicit unknown when appropriate. Preserve the original source so the recovery attempt cannot quietly substitute an invented answer. Log which correction was applied. If the same failure recurs, stop repeating the unchanged instruction and investigate offline. An effective recovery path produces information about the failure; an expensive loop merely generates more versions of the same uncertainty.

Treat external actions as a separate state machine

Sending an email, charging an account or updating a record introduces a question that text generation alone cannot answer: did the action complete? Store the action’s own identifier and use the system’s supported idempotency and status checks where available. Do not ask a new model response to infer success from a timeout. If completion is uncertain, preserve that state and resolve it through the action system before attempting another write. A retry budget limits repetition, but it does not by itself prevent duplicate side effects.

Review exhausted tasks before raising the limit

Group exhausted tasks by cause and inspect representative examples. If many failures arise from a schema that cannot express missing information, changing the schema may help more than allowing another call. If the issue is a provider incident, the right response may be service handling rather than prompt changes. Calculate cost per accepted task after any revision and compare it with the prior policy. Raising the attempt limit should be an evidence-based change with a clear expected benefit, not the default response to every unresolved task.