lesson 224 Data Formats & Analytical Workflows · advanced

Reproducible analysis

A result you can't reproduce can't be checked. Seed every random step, pass every setting in as a parameter, sort before you output, and fingerprint the result so two runs can be compared.

You'll be able to

Draw repeatable random samples with np.random.default_rng(seed) and sample(random_state=seed)
Write an analysis as a function of its inputs and parameters, with sorted, rounded output
Fingerprint a result with hashlib.sha256 of its CSV text and compare two runs

The idea

Run an analysis twice and get two answers, and nobody can tell which is right. The usual causes: random sampling without a seed; hidden settings (a threshold typed in the middle of the code, a file path that changed); and unstable order (rows in whatever order a dictionary, a join or a group happened to produce). The fixes are small: create random generators with a seed (np.random.default_rng(42), df.sample(n, random_state=42)), pass settings in as parameters, and sort the output by a key before saving it.

To check that two runs agree, compare a fingerprint: hashlib.sha256(text.encode()).hexdigest() turns any text (such as the CSV of the result) into a short code that changes if a single character changes.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Same seed, same sample

import numpy as np
import pandas as pd

trips = pd.DataFrame({"trip_id": [16126, 16127, 16128, 16129, 16130, 16131, 16132], "fare_amount": [40.8, 7.2, 17.2, 22.06, 5.1, 16.3, 42.9]})
first = np.random.default_rng(42).integers(0, 100, size=3).tolist()
again = np.random.default_rng(42).integers(0, 100, size=3).tolist()
print(first, first == again)
print(trips.sample(n=3, random_state=7)["trip_id"].tolist())
print(trips.sample(n=3, random_state=7)["trip_id"].tolist())

example-2.py · Hashing a DataFrame directly

import hashlib
import pandas as pd

result = pd.DataFrame({"payment_type_id": [0, 1], "avg_fare": [19.63, 22.46]})
print(hashlib.sha256(result).hexdigest())

example-3.py · An analysis that fingerprints its own output

import hashlib
import pandas as pd

def fare_summary(trips, min_fare):
    kept = trips[trips["fare_amount"] >= min_fare]
    summary = kept.groupby("payment_type_id", as_index=False)["fare_amount"].mean().round(2)
    return summary.sort_values("payment_type_id").reset_index(drop=True)

def fingerprint(table):
    return hashlib.sha256(table.to_csv(index=False).encode()).hexdigest()[:12]

trips = pd.DataFrame({"payment_type_id": [1, 1, 0, 0, 1], "fare_amount": [40.8, 7.2, 17.2, 22.06, 5.1]})
shuffled = trips.sample(frac=1, random_state=3)
a = fare_summary(trips, min_fare=6)
b = fare_summary(shuffled, min_fare=6)
print(a.to_dict("records"))
print(fingerprint(a) == fingerprint(b), fingerprint(a) == fingerprint(fare_summary(trips, min_fare=7.5)))

When to use it: every analysis someone else (or you, next month) may rerun. Mistakes to avoid: unseeded randomness; settings that live in the middle of the code instead of in parameters; output whose row order depends on how the data arrived; comparing floats from different machines character by character (round first); and assuming the same seed gives the same sample across library versions.

Review

Seed every random step: np.random.default_rng(seed), sample(random_state=seed); create the generator once.
Pass settings as parameters and sort output by a key, so the same inputs give the same result.
A SHA-256 of the result's CSV text is a fingerprint for comparing runs.
Next: write results others can read back exactly.

Practise this