lesson 159 Iterators & Generators · advanced

Lazy pipelines: generator expressions, islice and takewhile

Chained generators form a pipeline: read, parse, filter, transform, each step pulling one item at a time from the one before. Generator expressions write a step in one line, and itertools.islice and takewhile stop the whole pipeline as soon as you have enough.

You'll be able to

Write a generator expression and pass one straight to sum, max, any or all
Chain generator steps and predict the order in which their code runs
Stop early with islice, takewhile or next

The idea

A generator expression looks like a list comprehension in round brackets: (t["fare"] for t in trips if t["miles"] > 0). It makes a generator instead of a list, so nothing is computed until someone asks. Passed directly to a function, it doesn't even need its own brackets: sum(t["fare"] for t in trips).

Chain several generator steps and you get a pipeline. When the consumer asks for one value, the request travels back up the chain, and a single item flows down through every step before the next item is even read. So a pipeline over a 10 GB file holds one record at a time, and stops reading the moment you stop asking.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · One item through every step

def read(lines):
    for line in lines:
        print("read ", line)
        yield line

def parse(lines):
    for line in lines:
        trip_id, fare = line.split(",")
        yield int(trip_id), float(fare)

def expensive(trips, minimum):
    for trip_id, fare in trips:
        if fare >= minimum:
            print("keep ", trip_id)
            yield trip_id

source = ["16126,40.8", "16127,7.2", "16128,17.2", "16129,52.0"]
pipeline = expensive(parse(read(source)), 30)
print(next(pipeline))
print(next(pipeline))

itertools has the tools to stop a pipeline: `islice(it, n)` takes at most n items; `takewhile(test, it)` takes items while the test holds and stops at the first that fails. any and all (the Built-ins & Sorting module) stop as soon as they know the answer, and next(gen, default) takes just the first item.

example-2.py · Generator expressions and early stops

from itertools import islice, takewhile

trips = [
    {"trip_id": 16126, "miles": 9.2, "fare": 40.8},
    {"trip_id": 16127, "miles": 0.9, "fare": 7.2},
    {"trip_id": 16128, "miles": 3.17, "fare": 17.2},
    {"trip_id": 16129, "miles": 0.0, "fare": 3.0},
    {"trip_id": 16130, "miles": 18.4, "fare": 70.0},
]
per_mile = (t["fare"] / t["miles"] for t in trips if t["miles"] > 0)
print(type(per_mile).__name__)
print([f"{x:.2f}" for x in islice(per_mile, 2)])
print(sum(t["fare"] for t in trips if t["miles"] > 5))
print([t["trip_id"] for t in takewhile(lambda t: t["fare"] < 50, trips)])
print(any(t["miles"] == 0 for t in trips))
print(next((t["trip_id"] for t in trips if t["fare"] > 60), None))

When to use it: reading and filtering large inputs, searching for the first match, totals that don't need the values kept. Mistakes to avoid: writing [...] inside sum(...) or any(...) out of habit, which builds the whole list first and loses the early stop; using takewhile where you meant a filter (it stops at the first failure instead of skipping it); and expecting a generator expression to keep its values for a second pass.

Review

A generator expression is a lazy comprehension in round brackets; pass it straight to sum, any, all, max.
Chained generators move one item through every step before reading the next.
islice, takewhile, next(gen, default), any and all stop the pipeline as soon as they can.
Next: measuring what this saves in memory, and the one thing a generator can't do.

Practise this