lesson 239 Observability & Operations · advanced

Retry temporary failures with backoff

A locked database, a timed-out request or a busy server usually works a moment later. Retrying is right for those, and wrong for everything else. Good retries wait longer each time, add randomness, stop after a limit and re-raise the last error.

You'll be able to

Tell transient failures (timeouts, locks, 503) from permanent ones (bad input, 404) and retry only the first
Compute exponential backoff with a cap and full jitter, using an injected sleep and a seeded random generator
Give up after the last attempt and re-raise, and retry only operations that are safe to repeat

The idea

In the HTTP & API Clients module you retried HTTP requests. The same need appears everywhere a program talks to something it doesn't control: a database file locked by another writer, a queue that is momentarily full, an API answering 503 during a deploy. These transient failures go away by themselves. A permanent failure (invalid data, a missing table, a 404) will fail exactly the same way every time; retrying it only delays the error and adds load.

The policy has four parts. Which errors to retry: a specific list of exception types (and sometimes messages or status codes). How long to wait: exponential backoff, base × 2**attempt (0.5, 1, 2, 4 s…), so a struggling service gets more room each time, with a cap so waits don't grow without limit. Jitter: a random part of the wait, so a thousand clients that failed together don't all retry at the same instant. When to stop: after a fixed number of attempts, re-raise the last error so the caller knows.

Passing sleep in as a parameter keeps tests fast and exact: the examples record the waits instead of waiting.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Exponential backoff around a flaky call

def call_with_retry(operation, attempts, base, sleep):
    for attempt in range(attempts):
        try:
            return operation()
        except TimeoutError:
            if attempt == attempts - 1:
                raise
            delay = base * 2 ** attempt
            print(f"attempt {attempt + 1} timed out; waiting {delay} s")
            sleep(delay)

calls = []

def fetch_stars():
    calls.append("call")
    if len(calls) < 3:
        raise TimeoutError("star service did not answer")
    return {"lumen-labs/pyauth": 2}

waits = []
print(call_with_retry(fetch_stars, attempts=4, base=0.5, sleep=waits.append))
print("waited", waits, "after", len(calls), "calls")

When the attempts run out, the last error must reach the caller. Swallowing it (returning None) turns an outage into wrong data downstream:

example-2.py · Giving up re-raises the last error

def call_with_retry(operation, attempts, base, sleep):
    for attempt in range(attempts):
        try:
            return operation()
        except TimeoutError:
            if attempt == attempts - 1:
                raise
            print(f"attempt {attempt + 1} timed out; waiting {base * 2 ** attempt} s")
            sleep(base * 2 ** attempt)

def always_down():
    raise TimeoutError("star service did not answer")

call_with_retry(always_down, attempts=3, base=0.5, sleep=lambda seconds: None)

A reusable version is a decorator factory (see Decorators & Context Managers): the policy is set once, at the function's definition. This one retries only the exception types it is given, caps the wait, and uses full jitter: a random wait between 0 and the backoff value. A seeded random.Random makes the example repeatable; production code would use an unseeded one.

example-3.py · A retry decorator with a cap and jitter

import functools
import random
import sqlite3

def retry(attempts, base, cap, retry_on, sleep, rng):
    def decorate(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(attempts):
                try:
                    return func(*args, **kwargs)
                except retry_on:
                    if attempt == attempts - 1:
                        raise
                    sleep(rng.uniform(0, min(cap, base * 2 ** attempt)))
        return wrapper
    return decorate

waits = []
locks = [sqlite3.OperationalError("database is locked")] * 3

@retry(attempts=5, base=1, cap=3, retry_on=(sqlite3.OperationalError,), sleep=waits.append, rng=random.Random(42))
def save_star(repo):
    if locks:
        raise locks.pop()
    return f"saved star for {repo}"

print(save_star("tidewater/easystore"))
print([round(w, 2) for w in waits])

@retry(attempts=5, base=1, cap=3, retry_on=(sqlite3.OperationalError,), sleep=waits.append, rng=random.Random(42))
def parse_count(text):
    return int(text)

try:
    parse_count("two")
except ValueError:
    print("ValueError raised at once; waits still", len(waits))

One more rule: retry only what is safe to repeat. If a payment request timed out *after* the provider charged the card, a blind retry charges it twice. Reads and idempotent writes (the Reliable Data Pipelines module: upserts, writes with an idempotency key) are safe; anything else needs a key the server can use to recognise the repeat.

Mistakes to avoid: except Exception in a retry loop; constant or immediate retries; no cap and no limit; returning None when attempts run out; and retrying non-idempotent operations.

Review

Retry only transient failures (timeouts, locks, 429/503), and only operations that are safe to repeat.
Wait base × 2**attempt, capped, with jitter; honour a server's Retry-After.
Stop after a fixed number of attempts and re-raise the last error.
Next: when something does fail, the logs, metrics and traces are your evidence. Investigating a failure.

Practise this