lesson 200 Software Architecture · advanced

Integration tests: modules working together

Unit tests check each part alone. Integration tests run the real parts together through the program's entry point, faking only the outer edges (a temporary folder, an in-memory store), and catch the wiring bugs unit tests miss.

You'll be able to

Write an entry point that wires parsing, the core and output together
Test it end to end in a temporary folder with tempfile.TemporaryDirectory
Choose what to fake (outer edges only) and what to keep real (everything in between)

The idea

Each module of a stock importer can pass its own tests while the program as a whole fails: the parser returns (sku, change) tuples but the caller expects dictionaries, the report is written to the wrong file name, the header line is counted as stock. Integration tests catch this by running the real modules together, as a user would, through the entry point. They replace only the outer edges: the files go in a temporary folder (tempfile.TemporaryDirectory() creates one and deletes it after the with block), and remote services get an in-memory stand-in.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · A three-module importer, tested end to end

import importlib
import os
import tempfile

with open("parsing.py", "w", encoding="utf-8") as f:
    f.write("def parse_movements(text):\n    rows = []\n    for line in text.strip().splitlines()[1:]:\n        sku, change = line.split(',')\n        rows.append((sku.strip(), int(change)))\n    return rows\n")
with open("inventory.py", "w", encoding="utf-8") as f:
    f.write("class Inventory:\n    def __init__(self):\n        self.levels = {}\n    def apply(self, sku, change):\n        self.levels[sku] = self.levels.get(sku, 0) + change\n    def report(self):\n        return [f'{sku},{qty}' for sku, qty in sorted(self.levels.items())]\n")
with open("app.py", "w", encoding="utf-8") as f:
    f.write("from inventory import Inventory\nfrom parsing import parse_movements\n\ndef run_import(in_path, out_path):\n    with open(in_path, encoding='utf-8') as f:\n        movements = parse_movements(f.read())\n    stock = Inventory()\n    for sku, change in movements:\n        stock.apply(sku, change)\n    with open(out_path, 'w', encoding='utf-8') as f:\n        f.write('\\n'.join(stock.report()) + '\\n')\n    return len(movements)\n")
importlib.invalidate_caches()
from app import run_import

def test_import_end_to_end():
    with tempfile.TemporaryDirectory() as folder:
        source = os.path.join(folder, "moves.csv")
        target = os.path.join(folder, "stock.csv")
        with open(source, "w", encoding="utf-8") as f:
            f.write("sku,change\nSPO-11681,5\nHOM-22847,2\nSPO-11681,-1\n")
        assert run_import(source, target) == 3
        with open(target, encoding="utf-8") as f:
            assert f.read() == "HOM-22847,2\nSPO-11681,4\n"

test_import_end_to_end()
print("integration test passed")

With unittest (see Testing & Code Quality), integration tests sit next to unit tests and run with the same runner. A setUp builds the temporary folder (self.addCleanup(tmp.cleanup) removes it), and each test feeds a realistic input file: an empty file, a line with spaces, a negative movement. Keep integration tests fewer and broader than unit tests: they are slower and, when they fail, they say *something* in the chain broke; unit tests then say what.

example-2.py · Integration tests in a TestCase

import io
import os
import tempfile
import unittest

def parse_movements(text):
    rows = []
    for line in text.strip().splitlines()[1:]:
        sku, change = line.split(",")
        rows.append((sku.strip(), int(change)))
    return rows

def run_import(in_path, out_path):
    with open(in_path, encoding="utf-8") as f:
        movements = parse_movements(f.read())
    levels = {}
    for sku, change in movements:
        levels[sku] = levels.get(sku, 0) + change
    with open(out_path, "w", encoding="utf-8") as f:
        f.writelines(f"{sku},{qty}\n" for sku, qty in sorted(levels.items()))
    return len(movements)

class ImportTests(unittest.TestCase):
    def setUp(self):
        tmp = tempfile.TemporaryDirectory()
        self.addCleanup(tmp.cleanup)
        self.folder = tmp.name

    def run_with(self, text):
        source = os.path.join(self.folder, "in.csv")
        target = os.path.join(self.folder, "out.csv")
        with open(source, "w", encoding="utf-8") as f:
            f.write(text)
        count = run_import(source, target)
        with open(target, encoding="utf-8") as f:
            return count, f.read()

    def test_header_only(self):
        self.assertEqual(self.run_with("sku,change\n"), (0, ""))

    def test_spaces_and_negatives(self):
        self.assertEqual(self.run_with("sku,change\n GRO-10442 ,3\nGRO-10442,-3\n"), (2, "GRO-10442,0\n"))

result = unittest.TextTestRunner(stream=io.StringIO()).run(unittest.defaultTestLoader.loadTestsFromTestCase(ImportTests))
print(result.testsRun, len(result.failures), len(result.errors))

What to fake: only what is slow, external or uncontrollable: the network, the clock, third-party services, real user folders. Keep your own modules real, or the test proves nothing about how they fit. Mistakes to avoid: writing to real paths (tests that pass once and then fail, or delete real data); integration tests that depend on each other's leftovers; and replacing so much with fakes that only the fakes are tested.

Review

Integration tests run the real modules together through the entry point.
Fake only the outer edges: temporary folders for files, in-memory stand-ins for remote services.
Fewer and broader than unit tests; each catches wiring bugs unit tests can't see.
Next module: data analysis with NumPy and pandas.

Practise this