lesson 241 Application Delivery · advanced

Lock every dependency

`pandas>=2.2` in pyproject.toml says what your code accepts. A lock file says exactly what was tested: every package, including the ones pandas pulls in, at one version, with a hash of the file. Deploying from the lock means production runs what CI ran.

You'll be able to

Explain the difference between a requirement range (see Packaging & Databases) and a lock file's exact pins
Parse a lock file into normalised package names, versions and hashes
Verify a file's SHA-256 against the lock with hashlib, and report drift from the installed versions

The idea

In the Packaging & Databases module a project declared ranges: pandas>=2.2,<3. Ranges are right for a library and for your own intent, but they don't say what will be installed: pip picks the newest version that fits *on the day you install*. Build the same project next month and you may get pandas 2.3 and a new numpy underneath it, untested.

A lock file removes the choice. Generated by a tool (pip-compile, uv lock, poetry lock) from the declared ranges, it lists every package, the direct ones and everything they depend on, each pinned with == to one version, usually with a --hash=sha256:… of the exact file that was downloaded. Installing from it (pip install --require-hashes -r requirements.lock, a terminal command) reproduces the tested environment, and refuses a file whose hash differs, which also protects against a tampered download.

Package names are compared in normalised form: lower case, with runs of -, _ and . turned into one -, so python_dateutil and Python-DateUtil are the same package.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Read a lock file

import re

LOCK = """
# requirements.lock: generated from pyproject.toml, do not edit
numpy==2.1.3 --hash=sha256:2f10e1910e389422d852df6d9860305296fb716610ca6d234c5ca410b550b12b
pandas==2.2.3 --hash=sha256:53b4e41aa9e92b1cdec11db92d9d0f33db738f6ad4e4a942bbb11ec32886fdcd
python_dateutil==2.9.0.post0 --hash=sha256:bc5635f72a85baaef3dd498ba15635b4af6795cdab5e45093187d5f2ad5d4423

Requests==2.32.3 --hash=sha256:141615f1293c72aa2718a7be0ddaaeef2355c6f0b5e149cc32c8ba74b8a9e6e2
"""

def normalise(name):
    return re.sub(r"[-_.]+", "-", name).lower()

def read_lock(text):
    pins = {}
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        requirement, _, digest = line.partition(" --hash=sha256:")
        name, version = requirement.split("==")
        pins[normalise(name)] = (version, digest)
    return pins

pins = read_lock(LOCK)
for name, (version, digest) in pins.items():
    print(f"{name:16} {version:12} {digest[:12]}…")

A hand-written line with a range instead of a pin doesn't fit the lock format, and the parser says so loudly:

example-2.py · A range where a pin should be

line = "pandas>=2.2"
name, version = line.split("==")
print(name, version)

The hash is checked by hashing the file you actually have and comparing with the pinned value: hashlib.sha256(data).hexdigest() (you met hashing for reproducible runs in the Data Formats & Analytical Workflows module). And before a deploy, the environment itself can be checked against the lock: on a real machine the installed versions come from importlib.metadata (see Packaging & Databases); here they are passed in as a dictionary, so the comparison is the same whatever this page has installed.

example-3.py · Verify a download and look for drift

import hashlib

pins = {
    "numpy": ("2.1.3", "2f10e1910e389422d852df6d9860305296fb716610ca6d234c5ca410b550b12b"),
    "pandas": ("2.2.3", "53b4e41aa9e92b1cdec11db92d9d0f33db738f6ad4e4a942bbb11ec32886fdcd"),
}

def verify(path, expected):
    with open(path, "rb") as f:
        return hashlib.sha256(f.read()).hexdigest() == expected

with open("pandas-2.2.3.whl", "wb") as f:
    f.write(b"pandas-2.2.3 wheel bytes")
print("pandas file ok:", verify("pandas-2.2.3.whl", pins["pandas"][1]))
with open("pandas-2.2.3.whl", "ab") as f:
    f.write(b" plus something extra")
print("after tampering:", verify("pandas-2.2.3.whl", pins["pandas"][1]))

installed = {"numpy": "2.1.3", "pandas": "2.2.2", "rich": "13.9.4"}
for name, (version, _) in sorted(pins.items()):
    if installed.get(name) != version:
        print(f"drift: {name} locked {version}, installed {installed.get(name)}")
for name in sorted(set(installed) - set(pins)):
    print(f"drift: {name} installed but not locked")

When to use it: for every application you deploy; libraries keep ranges, applications ship locks. Regenerate the lock deliberately (and review the diff) when you upgrade. Mistakes to avoid: editing the lock by hand; ranges or bare names in it; comparing names without normalising; and treating a hash mismatch as a reason to retry the download instead of stopping.

Review

Declare ranges (pyproject.toml); deploy from a lock: every package, == one version, with a hash.
Normalise package names before comparing them.
Verify downloads with hashlib.sha256 and compare installed versions with the lock before a deploy.
Next: the lock fixes the Python packages; a Docker image fixes everything else around them.

Practise this