A temporary database failure often prompts a simple response: try again. That response can be appropriate, but it needs more information than an exception message. The application must know what failed, whether the original work could have committed and which decisions need to be repeated against a fresh state.
A retry can otherwise duplicate a business effect or replay a decision whose assumptions are no longer true. During an outage, uncontrolled retries can also multiply demand on the very service that is struggling to recover.
Reliable retry design therefore combines correctness and resource control. It defines a repeatable unit of work, recognises uncertain outcomes and limits the amount of additional work recovery is allowed to generate.
Classify the failure before repeating the operation
Consider a service application recording a material issue against a job. This is an illustrative example. A request can fail because its input is invalid, because another transaction created a conflict, because the connection broke or because the database temporarily lacks capacity.
Those cases do not justify the same response. Retrying an invalid code without changing the input is unlikely to help. A transaction rejected because of a transient concurrency conflict may be suitable for a fresh attempt. A broken connection during commit can leave the outcome uncertain.
Use the database and driver’s documented error classifications where available. Avoid interpreting arbitrary message text as a reliable protocol. Preserve enough diagnostic detail to distinguish a recognised retryable condition from an unexpected failure.
The operation’s meaning also matters. A read with no external effect can often be repeated more easily than a request to issue stock. Even a read can be costly or time-sensitive, so repeatability should not be confused with permission to retry indefinitely.
Create a deliberate classification policy: conditions that permit retry, conditions that require corrected input, conditions requiring outcome reconciliation and conditions that should fail visibly. Review the policy when changing drivers, database versions or transaction behaviour.
A timeout does not prove that nothing happened
Suppose the database commits the material issue, but the acknowledgement is lost before reaching the application. The caller sees a timeout even though the business effect is permanent. Repeating the request as a new issue can deduct the material twice.
The uncertainty concerns the outcome, not merely the connection. Opening a fresh connection does not answer whether the earlier transaction committed. The application needs a way to identify and reconcile the logical operation.
A stable operation identifier can support that process. If the issue request is recorded under identifier R, a later attempt can ask whether R was already accepted and retrieve its established outcome. The identifier must be reused for retries of the same logical request rather than regenerated on every attempt.
Define what happens if the caller reuses R with different content. Accepting a different quantity under the same identity can conceal a programming or user error. The receiving system should validate that the repeated request is consistent with the original operation’s defined identity and payload rules.
Some interfaces provide no reliable way to determine an ambiguous outcome. In that case, automatic repetition of a consequential write may be unsafe. Expose an unresolved state and use reconciliation rather than converting uncertainty into a duplicate effect.
Place duplicate protection around the actual effect
Checking an operation log before writing is not enough if the check and effect can race. Two workers may both see that R is absent and each apply the material issue. The record of the operation and its business change need an appropriate shared atomic boundary.
For a local database operation, that can mean establishing the operation identity and applying the related update within one transaction, supported by the necessary uniqueness and concurrency rules. The exact implementation depends on the database and command semantics.
Retain the result needed for a repeated caller. A response that merely says already processed may be insufficient if the caller needs the created record identifier or the accepted quantity. The retry should be able to receive a consistent account of the original outcome.
Define retention for the operation identity. If duplicate-protection records expire before old requests can be replayed, a delayed retry may be accepted as new work. Consider offline clients, queued messages, recovery procedures and restored application state when setting that period.
Do not assume that every operation can be made repeatable by attaching a token. The token must participate in the actual write boundary, and external effects need their own supported handling. A local operation log cannot by itself make an unrelated remote action atomic with the database change.
Retry the decision, not only the last statement
A transaction may read availability, choose a resource and write a reservation. If the database rejects that transaction because its concurrent execution could not be completed safely, retrying only the final insert can reuse a decision based on an obsolete state.
The new attempt should generally rerun the complete transaction logic required to make the decision, including relevant reads and calculations. PostgreSQL’s serialization-failure guidance explicitly discusses retrying the complete transaction rather than selected statements alone.
Keep the retryable unit separate from the surrounding user interaction. A person need not re-enter the request for every transient transaction conflict, but the server must reevaluate the current state before making a fresh commitment.
Decide which inputs remain fixed. The user’s requested quantity may remain constant, while the selected stock location is recalculated. Random choices, generated identifiers and time-dependent decisions need deliberate treatment so that retrying does not accidentally create a different logical request.
The result of a fresh attempt may legitimately differ from the original expectation. Stock that was available earlier may no longer be available. Report that business outcome clearly rather than treating every unsuccessful retry as another transient technical problem.
Read-only work can need a similar boundary. If a report combines several queries intended to observe one consistent population, retrying just the failed query in a new transaction may mix different snapshots. Decide whether the report can tolerate that mixture or must restart its complete read operation. The absence of writes makes duplicate business effects less concerning, but it does not remove the need for a coherent result.
Keep external effects out of blindly repeated transaction bodies
If a retryable transaction sends an email or calls another service before it commits, rerunning the transaction can repeat that external effect even when the database rolls back. Local transactional protection does not extend automatically to those operations.
Separate the database decision from follow-up delivery through a suitable reliable design. One approach records the intended outgoing action together with the business update and handles delivery afterwards. The delivery process still needs its own duplicate and outcome rules.
Do not hide external work inside a helper function without considering its retry semantics. A method that appears to calculate a value may also create a remote reservation or write an audit record elsewhere. Review the full set of effects in the repeated unit.
Logging also deserves thought. Diagnostic attempt logs may intentionally contain several entries for one logical operation, while a business event should represent the accepted outcome according to its defined semantics. Distinguish attempt identity from operation identity so that later analysis does not count retries as separate transactions.
Where a remote action must occur before the database decision can finish, model the coordination explicitly. The process may require durable intermediate state and compensation or reconciliation. A generic retry wrapper cannot supply those business semantics on its own.
Use one overall deadline, not a fresh budget for every attempt
A request with three attempts can take far longer than expected if each attempt receives the full user-facing timeout. Connection acquisition, statement execution, backoff and cleanup all consume time. The caller needs an overall deadline that bounds the complete operation.
Before each attempt, calculate whether enough time remains for useful work. Starting a new transaction with only a tiny fraction of the budget left can create additional load and uncertainty without a realistic chance of returning a result.
Distinguish several timeout scopes. Waiting for a pooled connection, establishing a connection, waiting for a lock and executing a statement are different activities. A timeout in one scope does not establish that another activity stopped at the same moment.
Cancellation needs verification too. The application may stop waiting while the database continues processing until a cancellation request is received and acted upon. The retry policy must account for the actual driver and server behaviour rather than assuming that a client deadline rolls back all work instantly.
Communicate the final state accurately. If the operation definitively failed, say so. If its outcome is still being reconciled, return or record that uncertainty through the application’s supported workflow. Do not present a timeout as proof that the business request had no effect.
Spread retries and limit their total volume
When many clients fail together and retry on the same fixed schedule, they can create repeated bursts of load. Backoff increases the delay between attempts, while jitter varies timing so that clients do not all return at once.
The precise schedule should fit the service and request deadline. It is not a substitute for limiting concurrent work. A large population of requests can still overwhelm a recovering database even when each individual request waits between attempts.
Use bounded attempts and an appropriate retry budget. Track the additional load generated by retries relative to ordinary requests. If recovery traffic dominates, continuing to repeat work may delay recovery and increase the number of callers reaching their deadlines.
Choose a layer responsible for retrying the logical operation. If a browser, gateway, service and database helper each retry independently, one user request can expand into many backend attempts. With three attempts at each of four nested layers, the theoretical maximum is 81 lowest-level attempts.
AWS’s guidance on timeouts, retries and jitter discusses these load-amplification concerns. The operational lesson is to coordinate retry ownership and budget across the request path, rather than adding another local retry loop wherever an error is observed.
Release failed resources before waiting
A retry delay should not unnecessarily hold a database connection, open transaction or other scarce resource. Complete the required rollback or disposal, then wait according to the policy. Otherwise idle retrying requests can occupy the resources needed for useful recovery work.
A failed connection may be unusable and should not be returned to the pool as if it were healthy. Follow the driver’s supported invalidation and cleanup behaviour. Reusing a broken resource can turn every retry into the same immediate failure.
Transaction state also matters. Some errors leave a transaction requiring rollback before any further statements can proceed. Retrying a statement inside that failed transaction without resetting the state is not a fresh attempt.
Make cleanup robust to its own failures. If rollback or close raises another error, preserve the original failure context while ensuring the resource is not silently reused in an unknown state. Resource ownership should be clear enough that two layers do not each assume the other completed cleanup.
Test abandoned requests. A caller may disconnect during backoff or cancel while an attempt is running. The system should stop unnecessary future attempts where appropriate and still reconcile any consequential work whose outcome is uncertain.
Observe logical operations and attempts separately
An operation that succeeds after two retries is one accepted business action and three technical attempts. Monitoring only the final success rate can hide increasing contention or connectivity problems. Counting every attempt as a business action can exaggerate activity.
Record a logical operation identifier and an attempt number, together with the recognised failure category and elapsed time. Avoid placing sensitive request payloads into routine logs merely to make retries traceable.
Measure first-attempt success, eventual success, retry volume, exhausted deadlines and unresolved outcomes. These measures answer different questions. A high eventual success rate may still be unacceptable if most users wait through repeated delays.
Examine concentrations by operation type and dependency. Frequent serialization retries may indicate a contested decision or transaction scope that needs review. Frequent connection failures may point to pool or network behaviour. The retry mechanism should reveal the underlying issue rather than permanently conceal it.
Give unresolved outcomes an operational owner. A request that might have committed is not adequately handled by a generic error counter. The organisation needs a way to establish the business result and communicate it to the caller without creating another independent request.
Test failures at the boundaries that create uncertainty
Inject failure before a transaction starts, during its work, after commit but before acknowledgement and while the result is being recorded by the caller. Each boundary tests a different part of the design.
Repeat the same operation concurrently and verify one logical effect. Reuse its identifier with inconsistent content and confirm that the mismatch is detected. Replay an old request near the duplicate-protection retention boundary to test the documented policy.
For transaction-conflict retries, change the data used by the original decision before the next attempt. Confirm that the decision is recalculated and that an obsolete resource choice is not reused blindly.
Test a broad outage with many callers. Observe retry timing, pool occupancy and the recovery queue. Confirm that deadlines and cancellation stop unnecessary work and that the system can recover without an expanding wave of repeated attempts.
Retry logic is trustworthy when it repeats only appropriate work, preserves the identity of the business request and makes uncertain outcomes visible. Trying again is a recovery action whose safety depends on those guarantees.
Source basis and further reading
The Retryer pattern in the source collection’s Data Access Patterns motivates separating retry policy from operation-specific failure and recovery logic. This article uses an original material-issue example and does not adopt the book’s historical product-specific recovery actions.
PostgreSQL documents serialization-failure handling. AWS discusses timeouts, retries and backoff with jitter. Verify failure codes, transaction state and cancellation behaviour against the actual database and driver before implementing a policy.
