lesson 245 Application Delivery · advanced

Check a deployment is ready

A new version can pass its tests and still fail in production: the database wasn't migrated, a setting is missing, a folder isn't writable. Readiness checks run against the real environment before traffic moves, and database migrations bring the schema to the version the code expects.

You'll be able to

Write readiness checks that each report pass or fail with a reason, and a runner that runs all of them and decides
Read a database's schema version (PRAGMA user_version) and its tables, and compare them with what the code needs
Apply pending migrations in order, each in its own transaction, so a failed step leaves the database as it was

The idea

Tests (L03) prove the code works on test data. A readiness check proves *this deployment* can work: the settings load (L04), the database answers and has the schema this version expects, the folders it writes to exist. Platforms call such a check before sending traffic to a new instance, and a release script runs it before switching versions. Two rules make it useful: run every check, even after one fails, so the report shows all problems at once; and give each failure a reason a person can act on.

The most common reason a deploy fails is the database. New code expects a new column; the database still has the old schema. Projects number their schema changes (migrations) and record the current number in the database; SQLite has a built-in slot for it, PRAGMA user_version. Deploying runs the migrations the database hasn't had yet, in order, and the readiness check confirms the number matches what the code expects.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Run every check and decide

import os
import sqlite3

class CheckFailed(Exception):
    pass

def check_settings():
    if not os.environ.get("DATABASE_URL"):
        raise CheckFailed("DATABASE_URL is not set")
    return "settings loaded"

def check_schema():
    conn = sqlite3.connect(":memory:")
    version = conn.execute("PRAGMA user_version").fetchone()[0]
    if version != 3:
        raise CheckFailed(f"schema version {version}, this release needs 3")
    return "schema version 3"

def check_exports_folder():
    os.makedirs("exports", exist_ok=True)
    with open("exports/.ready", "w", encoding="utf-8") as f:
        f.write("ok")
    return "exports folder writable"

def readiness(checks):
    report = []
    for name, check in checks:
        try:
            report.append((name, True, check()))
        except CheckFailed as err:
            report.append((name, False, str(err)))
        except Exception as err:
            report.append((name, False, f"{type(err).__name__}: {err}"))
    return report, all(ok for _, ok, _ in report)

os.environ.pop("DATABASE_URL", None)
report, ready = readiness([("settings", check_settings), ("schema", check_schema), ("exports", check_exports_folder)])
for name, ok, detail in report:
    print("PASS" if ok else "FAIL", name, "-", detail)
print("ready to deploy:", ready)

What happens if the new code goes live on the old schema? The first query that needs the new column fails, in front of a user:

example-2.py · New code, old schema

import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (order_id TEXT PRIMARY KEY, total REAL)")
conn.execute("INSERT INTO orders VALUES ('O100282', 35.84)")

# This release reports orders by status, added in schema version 2.
print(conn.execute("SELECT status, COUNT(*) FROM orders GROUP BY status").fetchall())

Migrations are a list of SQL steps keyed by version number. The connection below is opened with isolation_level=None, so the code controls transactions itself: BEGIN, the step's statements, the new version number, COMMIT. SQLite's schema changes are transactional, so if a statement fails, ROLLBACK undoes the whole step, including the version number. Running the migration again later only applies what is still missing, which makes it idempotent (see Reliable Data Pipelines).

example-3.py · Migrate to the version the code expects

import sqlite3

MIGRATIONS = {
    1: ["CREATE TABLE orders (order_id TEXT PRIMARY KEY, total REAL)"],
    2: ["ALTER TABLE orders ADD COLUMN status TEXT NOT NULL DEFAULT 'processing'"],
    3: ["CREATE TABLE payments (payment_id TEXT PRIMARY KEY, order_id TEXT, amount REAL)",
        "CREATE INDEX payments_order ON payments (order_id)"],
}

def migrate(conn, migrations):
    current = conn.execute("PRAGMA user_version").fetchone()[0]
    applied = []
    for version in sorted(v for v in migrations if v > current):
        conn.execute("BEGIN")
        try:
            for statement in migrations[version]:
                conn.execute(statement)
            conn.execute(f"PRAGMA user_version = {int(version)}")
            conn.execute("COMMIT")
        except sqlite3.Error:
            conn.execute("ROLLBACK")
            raise
        applied.append(version)
    return applied

conn = sqlite3.connect(":memory:", isolation_level=None)
print("applied", migrate(conn, MIGRATIONS))
print("applied", migrate(conn, MIGRATIONS))
print("version", conn.execute("PRAGMA user_version").fetchone()[0])
print([row[0] for row in conn.execute("SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name")])

When to use it: as the last gate of every release, and as the health endpoint a platform polls. Mistakes to avoid: stopping at the first failed check; checks that only say "failed"; editing an applied migration instead of adding a new one; migrations outside a transaction (a half-applied step leaves a schema no version describes); and deploying code before migrating the database it needs.

Review

Readiness checks test the real deployment: settings, schema version and tables, writable folders. Run them all and give reasons.
Migrations are numbered steps; PRAGMA user_version records how far a SQLite database has got.
Apply each pending step in its own transaction with its version number; a failure rolls back and re-raises.
Next module: the Production Readiness Capstone module brings everything together in one production-ready pipeline.

Practise this