lesson 157 Iterators & Generators · advanced

Generator functions: yield one value at a time

A function that contains yield doesn't run when you call it. It returns a generator, an iterator whose body runs only when the next value is asked for, pausing at each yield. That makes reading a large file record by record as easy as writing a loop.

You'll be able to

Explain what calling a generator function returns, and when its body runs
Write generators that read, convert and filter records one at a time
Stop early and know which work was never done

The idea

Writing a class with __next__ and a position for every sequence is a lot of ceremony. A generator function does the same job: any function whose body contains yield returns a generator when called. Nothing in the body runs yet. Each next() runs the body until the next yield, hands out that value and pauses right there, local variables and all. When the body finishes, the generator raises StopIteration for you.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · When a generator's body runs

def numbered(lines):
    print("generator starts")
    for number, line in enumerate(lines, start=1):
        print("  reading line", number)
        yield number, line.strip()
    print("generator finished")

events = numbered(["PushEvent\n", "WatchEvent\n", "ForkEvent\n"])
print(type(events).__name__)
print(next(events))
for item in events:
    print(item)

The pattern shines on files. A generator can open a file, walk it line by line, and yield only the records you asked for. The caller sees a plain iterator and never deals with the file. Because the body runs on demand, a caller that stops early never reads the rest of the file.

example-2.py · Events of one type from a JSONL file

import json
from pathlib import Path

events = [
    {"id": "48000004214", "type": "CreateEvent", "repo": {"name": "cobalt-tools/deeplint"}},
    {"id": "48000002730", "type": "WatchEvent", "repo": {"name": "tidewater/smartqueue"}},
    {"id": "48000001239", "type": "PullRequestReviewEvent", "repo": {"name": "tidewater/smartqueue"}},
    {"id": "48000003301", "type": "WatchEvent", "repo": {"name": "cobalt-tools/deeplint"}},
]
Path("events.jsonl").write_text("\n".join([json.dumps(e) for e in events]) + "\n", encoding="utf-8")

def events_of_type(path, wanted):
    with open(path, encoding="utf-8") as f:
        for line in f:
            event = json.loads(line)
            if event["type"] == wanted:
                yield event

stars = events_of_type("events.jsonl", "WatchEvent")
first = next(stars)
print(first["repo"]["name"])
print([event["id"] for event in stars])

When to use it: whenever a function would build a list only so that someone can loop over it once: reading records, converting rows, walking pages. Mistakes to avoid: writing return value inside the loop (the function returns once, it is not a generator unless it contains yield); expecting the body to run when the function is called; and forgetting that, like any iterator, a generator is used up after one pass.

Review

A function with yield returns a generator; its body runs only when values are asked for, pausing at each yield.
Generators turn "read, convert, filter" loops into iterators that callers can consume one record at a time.
Stopping early means the rest of the work is never done; a generator is still used up after one pass.
Next: yield from, which hands on every value of another iterable, for chaining and recursion.

Practise this