lesson 64 Comprehensions · intermediate

Flatten and combine with two for clauses

Two `for` clauses in one comprehension work like a loop inside a loop: they flatten lists of lists, or pair every item of one list with every item of another.

You'll be able to

Flatten a list of lists with [x for row in rows for x in row]
Put the for clauses in the same order as nested loops, outer first
Tell a flat result (two for clauses) from a list of lists (a comprehension inside the expression)

The idea

Data often arrives in groups: readings grouped by week, order lines grouped by order, tags grouped by question. To work on every value at once you need them in one flat list. With loops that is a loop inside a loop (see Loops & Iteration): for week in weeks:, and inside it for t in week:, and inside that temps.append(t).

The comprehension keeps the two for lines in the same order, outer first, and puts the value in front: [t for week in weeks for t in week].

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Three weeks of readings, one list

weeks = [
    [6.7, 7.2, 3.3],
    [7.2, 3.9],
    [-2.1, 0.5, 1.1],
]
temps = [t for week in weeks for t in week]
print(temps)
print(len(weeks), len(temps))

Reading the clauses in the order of the sentence (*t for t in week, for week in weeks*) is tempting, but Python runs the first for first. Swapped, the first clause uses week before anything has created it.

example-2.py · The for clauses swapped

weeks = [[6.7, 7.2], [3.9]]
temps = [t for t in week for week in weeks]
print(temps)

The inner list doesn't have to be a variable of its own: it can be a field of each record. Every order below has a list of lines, and the comprehension produces one pair per line. An order with no lines contributes nothing.

example-3.py · Every line of every order

def order_lines(orders):
    return [(order["order_id"], line["sku"]) for order in orders for line in order["lines"]]

orders = [
    {"order_id": "O100282", "lines": [{"sku": "SPO-11681", "qty": 2}, {"sku": "HOM-22847", "qty": 1}]},
    {"order_id": "O100918", "lines": []},
    {"order_id": "O101219", "lines": [{"sku": "ELE-24781", "qty": 1}]},
]
for pair in order_lines(orders):
    print(pair)

When the second for walks a list that doesn't depend on the first, you get every combination: every size with every colour, every row with every seat letter.

A different shape is a comprehension inside the expression: [[cell.strip() for cell in line.split(",")] for line in lines] builds a list of lists, one inner list per line, instead of one flat list.

When to use it: flattening one level of nesting, or listing combinations. Mistakes to avoid: swapping the clauses, and going deeper than two: three for clauses are hard to read, and a loop with names for each level is clearer.

Review

[x for row in rows for x in row] flattens: the for clauses are in the order of nested loops, outer first.
Two independent for clauses list every combination.
A comprehension inside the value keeps the nesting: one inner list per outer item.
Next: an if at the end that keeps only some items, and when a plain loop is the better choice.