lesson 125 Testing & Code Quality · intermediate

Edge cases that catch bugs

Most bugs live at the edges: an empty list, one item, the exact boundary, a repeated value. A good test suite is one that fails when the code is broken, and you can check that by running it against deliberately broken versions.

You'll be able to

List the edge cases for a function: empty, one item, boundaries, duplicates, order, invalid input
Decide and test what a function should do at an edge (return a value or raise)
Judge a test suite by running it against deliberately broken versions of the code

The idea

Tests written from a typical example tend to pass: the typical case is the one the code was written for. Bugs hide where the code's assumptions break:

empty input (no orders yet, an empty file) and a single item;
boundaries: exactly at a limit, one below, one above; the first and last page;
duplicates and ties; input in an unexpected order;
invalid input: negative, missing, the wrong type.

An edge case often forces a decision the specification forgot. What *is* the average rating of a product with no ratings? Decide (here: None), write it down, and test it.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · A typical test passes; an edge case doesn't

import unittest

def average_rating(ratings):
    return sum(ratings) / len(ratings)

class TestTypicalOnly(unittest.TestCase):
    def test_three_ratings(self):
        self.assertAlmostEqual(average_rating([5, 4, 3]), 4.0)

class TestWithEdges(unittest.TestCase):
    def test_three_ratings(self):
        self.assertAlmostEqual(average_rating([5, 4, 3]), 4.0)

    def test_one_rating(self):
        self.assertEqual(average_rating([2]), 2.0)

    def test_no_ratings_yet(self):
        self.assertIsNone(average_rating([]))

def run(case):
    result = unittest.TestResult()
    unittest.defaultTestLoader.loadTestsFromTestCase(case).run(result)
    print("ran", result.testsRun, "| failures", len(result.failures), "| errors", len(result.errors))
    for test, details in result.failures + result.errors:
        print("  " + test.id().split(".", 1)[1], "->", details.strip().splitlines()[-1])

run(TestTypicalOnly)
run(TestWithEdges)

How do you know a suite is good enough? Passing proves little: an empty suite passes too. The useful question is the reverse: would it fail if the code were wrong? You can check that for real. Write a few deliberately broken versions of the function (each with one typical mistake), run the suite against each, and see which tests catch it. A broken version that no test catches shows exactly which case is missing. (Tools such as mutmut automate this on a computer; it is called *mutation testing*.)

example-2.py · Which tests catch which broken version

import unittest

def page_correct(items, number, size):
    start = (number - 1) * size
    return items[start:start + size]

def page_skips_first(items, number, size):
    return items[number * size:(number + 1) * size]

def page_drops_partial(items, number, size):
    start = (number - 1) * size
    chunk = items[start:start + size]
    if len(chunk) < size:
        return []
    return chunk

ORDERS = ["O1", "O2", "O3", "O4", "O5", "O6", "O7"]

class TestPage(unittest.TestCase):
    def test_first_page(self):
        self.assertEqual(page(ORDERS, 1, 3), ["O1", "O2", "O3"])

    def test_last_page_is_partial(self):
        self.assertEqual(page(ORDERS, 3, 3), ["O7"])

    def test_past_the_end(self):
        self.assertEqual(page(ORDERS, 4, 3), [])

    def test_no_orders(self):
        self.assertEqual(page([], 1, 3), [])

for version in [page_correct, page_skips_first, page_drops_partial]:
    page = version
    result = unittest.TestResult()
    unittest.defaultTestLoader.loadTestsFromTestCase(TestPage).run(result)
    caught = []
    for test, details in result.failures + result.errors:
        caught.append(test.id().split(".")[-1])
    print(version.__name__, "->", caught)

When to use it: for every function you test. Before writing tests, list the edges of its input; after writing them, ask which plausible mistake would still pass.

Mistakes to avoid: testing only the example from the specification; skipping the empty case because "it never happens"; and edge-case tests that only check that *something* is returned (assertIsNotNone) instead of the exact expected value.

Review

Bugs cluster at edges: empty, one item, boundaries, duplicates, order, invalid input.
An edge case often needs a decision (return None? raise?): make it, then test exactly that.
Judge a suite by the broken versions it catches, not by how many tests it has.
Next module: classes, so your own objects can bundle data with the functions that work on it.

Practise this