lesson 243 Application Delivery · advanced

Let the tests decide the build

Continuous integration runs the same commands on every push: install from the lock, lint, run the tests. Its whole verdict is one number, the exit code. Here you run a test suite from Python, turn the result into that exit code, and read a CI log to find what failed.

Read and diagnose: this can't run in the browser, so you'll read real code and output and answer questions about it.

You'll be able to

Explain what a CI pipeline does and why its result is an exit code
Load and run TestCase classes with unittest, capturing the report, and map the result to 0 or 1
Read a CI log: find the step that failed and the tests that failed in it

The idea

Continuous integration (CI) is a service (GitHub Actions, GitLab CI, Jenkins) that, on every push, starts a clean machine and runs a list of steps written in a config file: check out the code, pip install --require-hashes -r requirements.lock, run a linter, python -m unittest. Each step is a command, and each command ends with an exit code: 0 means success, anything else means failure. The first step that exits non-zero stops the pipeline and marks the push red, and nothing gets deployed.

So the contract between your tests and CI is tiny: *exit with 0 if and only if every test passed.* python -m unittest does that for you. Programs that run tests themselves (a release script, the readiness checks in L05) do it with unittest's API: load the tests, run them with a runner whose report goes to a stream you choose, and look at the result.

About this lesson: the CI service itself can't run in this page. The test runs below are real; the CI log in the second example is sample text in the format GitHub Actions prints, read by Python.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Run a suite and decide the exit code

import io
import unittest

def price_with_tax(price, rate):
    return round(price * (1 + rate), 2)

class PricingTest(unittest.TestCase):
    def test_adds_tax(self):
        self.assertEqual(price_with_tax(10.0, 0.07), 10.7)

    def test_zero_rate(self):
        self.assertEqual(price_with_tax(31.0, 0), 31.0)

    def test_rounds_to_cents(self):
        self.assertEqual(price_with_tax(7.49, 0.0825), 8.11)

def run_ci(case):
    suite = unittest.defaultTestLoader.loadTestsFromTestCase(case)
    report = io.StringIO()
    result = unittest.TextTestRunner(stream=report, verbosity=0).run(suite)
    return result, 0 if result.wasSuccessful() else 1

result, code = run_ci(PricingTest)
print("ran", result.testsRun, "exit code", code)

def price_with_tax(price, rate):
    return price * (1 + rate)

result, code = run_ci(PricingTest)
print("ran", result.testsRun, "failures", len(result.failures), "exit code", code)
print("failed:", [test.id().split(".")[-1] for test, _ in result.failures])

Why a runner instead of calling test functions directly? Because a failing assert raises and stops everything after it: you learn about the first failure only, and nothing reports how many tests ran.

example-2.py · Calling tests by hand stops at the first failure

def price_with_tax(price, rate):
    return price * (1 + rate)

def test_adds_tax():
    assert price_with_tax(10.0, 0.07) == 10.7

def test_zero_rate():
    assert price_with_tax(31.0, 0) == 31.0

test_adds_tax()
test_zero_rate()
print("all tests passed")

When a pipeline is red, the log tells you why. GitHub Actions marks each step with ##[group]Run <command> and a failing step with ##[error]Process completed with exit code N.; unittest's own lines name each failing test as FAIL: test_name (module.Class.test_name) (or ERROR: for an unexpected exception). Reading that by eye in a long log is slow; a few lines of Python find it.

example-3.py · Find the failing step in a CI log

LOG = """##[group]Run actions/checkout@v4
##[endgroup]
##[group]Run pip install --require-hashes -r requirements.lock
Successfully installed numpy-2.1.3 pandas-2.2.3
##[endgroup]
##[group]Run python -m unittest
..F.E
======================================================================
FAIL: test_rounds_to_cents (tests.test_pricing.PricingTest.test_rounds_to_cents)
----------------------------------------------------------------------
AssertionError: 8.107925 != 8.11
======================================================================
ERROR: test_loads_rates (tests.test_rates.RatesTest.test_loads_rates)
----------------------------------------------------------------------
FileNotFoundError: [Errno 2] No such file or directory: 'rates.json'
Ran 5 tests in 0.031s
FAILED (failures=1, errors=1)
##[error]Process completed with exit code 1.
"""

step = None
failed_step = None
tests = []
for line in LOG.splitlines():
    if line.startswith("##[group]Run "):
        step = line.removeprefix("##[group]Run ")
    elif line.startswith("##[error]Process completed with exit code"):
        failed_step = step
    elif line.startswith(("FAIL: ", "ERROR: ")):
        tests.append(line.split()[1])
print("failed step:", failed_step)
print("tests:", tests)

When to use it: on every push and before every deploy; a repository with tests but no CI has tests nobody runs. Mistakes to avoid: a test script that prints failures but exits 0 (the pipeline stays green); tests that depend on the CI machine's leftovers instead of fresh fixtures (see Testing & Code Quality); installing with ranges instead of the lock (L01), so CI tests different versions than production runs; and ignoring ERROR lines because they aren't FAIL.

Review

CI runs the same steps on every push; each step's exit code (0 = success) decides whether the pipeline continues.
Run tests with unittest's loader and runner, report to a stream, and turn wasSuccessful() into 0 or 1.
In a red log, find the first failing step and the FAIL:/ERROR: tests inside it.
Next: the same build runs in several environments; configuration tells it which.

Practise this

Read unittest's summary line