lesson 168 Functional & Dynamic Python · advanced

Composition: build pipelines from small functions

Composing functions feeds each one's result into the next. A pipeline written as a list of small, pure steps is easy to read, reorder, test and configure, and functools.reduce applies the list in one expression.

You'll be able to

Write compose (right to left) and pipeline (left to right) for one-argument functions
Apply a list of steps to a value with a loop or functools.reduce
Keep each step small and pure so the pipeline can be rearranged and tested

The idea

Cleaning a product name from Open Food Facts takes several small steps: strip the ends, collapse repeated spaces, fix the capitals. Writing them as nested calls, title(collapse(strip(name))), reads inside out. Composition builds one function out of several, so the steps can be named, listed and reused.

In mathematics, compose(f, g)(x) is f(g(x)): the last function runs first. A pipeline lists the steps in the order they run, which most people find easier to read. Both are a loop that feeds each result into the next function, and functools.reduce(function, items, start) is exactly that loop: it carries a value through the items.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · compose and pipeline

from functools import reduce

def compose(*funcs):
    def composed(value):
        for func in reversed(funcs):
            value = func(value)
        return value
    return composed

def pipeline(*steps):
    def run(value):
        return reduce(lambda acc, step: step(acc), steps, value)
    return run

def collapse_spaces(text):
    return " ".join(text.split())

clean = pipeline(str.strip, collapse_spaces, str.title)
same = compose(str.title, collapse_spaces, str.strip)
print(clean("  super   smooth porridge "))
print(same("  super   smooth porridge "))
print(pipeline()("unchanged"))

Steps don't have to work on text. Over a list of records, each step can take a list and return a new one (filter, transform, sort), and the pipeline becomes a small data-processing plan. Because each step is pure, the order is easy to change and each step can be tested on its own.

example-2.py · A pipeline over product records

from functools import partial, reduce

def run_steps(rows, steps):
    return reduce(lambda acc, step: step(acc), steps, rows)

def with_name(rows):
    return [row for row in rows if row.get("product_name")]

def tidy_brands(rows):
    return [{**row, "brands": [b.strip() for b in row["brands"]]} for row in rows]

def sugar_at_most(rows, limit):
    return [row for row in rows if row["sugars_100g"] <= limit]

products = [
    {"code": "5010029219494", "product_name": "Super smooth porridge", "brands": ["Weetabix", " Ready break"], "sugars_100g": 1},
    {"code": "7613032192570", "product_name": "NESTLE LION Céréales 480g", "brands": ["Nestlé", " Lion"], "sugars_100g": 25},
    {"code": "0000000000000", "product_name": "", "brands": [], "sugars_100g": 3},
]
steps = [with_name, tidy_brands, partial(sugar_at_most, limit=10)]
for row in run_steps(products, steps):
    print(row["code"], row["brands"])
print(len(products), products[0]["brands"])

When to use it: text clean-up, record cleaning, transformations configured by settings, anything you would describe as "first this, then that". Mistakes to avoid: applying the steps in the wrong order (compose runs right to left); steps that change their input, which makes the order matter in surprising ways; and steps with different shapes (a step that returns None breaks every step after it).

Review

compose(f, g)(x) is f(g(x)); a pipeline applies its steps in the order listed.
functools.reduce(lambda acc, step: step(acc), steps, start) carries a value through a list of steps.
Small pure steps with one shape (same kind in, same kind out) can be reordered and configured safely.
Next: looking inside objects and functions while the program runs.