lesson 158 Iterators & Generators · advanced

yield from: chain and recurse

yield from hands on every value of another iterable as if the generator had yielded them itself. It chains several sources into one stream and makes recursive generators, such as walking a tree of folders or comments, short and clear.

You'll be able to

Replace a for x in other: yield x loop with yield from other
Chain several sources into one stream
Write recursive generators that walk nested data

The idea

Often a generator's job is to pass on what another iterable produces: all trips from several days, all files in several folders. yield from other does that in one line: it yields each value of other in turn, then carries on with the next line. It is equivalent to for value in other: yield value (and also passes through some generator details that matter in advanced code).

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · One stream from several days

def trips_from(*days):
    for day in days:
        yield from day

monday = [16126, 16127]
tuesday = [17001]
wednesday = []
thursday = [18210, 18211]
print(list(trips_from(monday, tuesday, wednesday, thursday)))

The bigger win is recursion (see Advanced Functions & Recursion). Nested data, such as an issue thread where every comment can have replies, is naturally walked by a generator that yields the current item and then *delegates* to itself for each child. Without yield from, the recursive call would create a generator and throw it away.

example-2.py · Walk a comment thread

thread = {"author": "data_fox514", "text": "Parser drops trailing zeros", "replies": [
    {"author": "stackkoi785", "text": "Can reproduce on 2.1", "replies": [
        {"author": "data_fox514", "text": "Same on main", "replies": []},
    ]},
    {"author": "hack_lynx150", "text": "Fix in #415", "replies": []},
]}

def walk(comment, depth=0):
    yield depth, comment["author"], comment["text"]
    for reply in comment["replies"]:
        yield from walk(reply, depth + 1)

for depth, author, text in walk(thread):
    print("  " * depth + f"{author}: {text}")
print(sum([1 for _ in walk(thread)]), "comments")

When to use it: to combine several iterables into one stream, and in recursive generators over trees (folders, comments, nested JSON). Mistakes to avoid: calling the recursive generator without yield from (its values are lost); writing yield walk(child), which yields the generator object itself instead of its values; and recursing on text: a string is iterable too, one character at a time.

Review

yield from other yields every value of other, then carries on.
It chains sources into one stream and lets a recursive generator pass on everything its children yield.
Without it, a recursive call is a generator nobody iterates; yield child_gen yields the generator object itself.
Next: building whole pipelines of lazy steps, with generator expressions and itertools.

Practise this