lesson 160 Iterators & Generators · advanced

Memory: what laziness saves and what it costs

A list holds every value at once; a generator holds one. tracemalloc measures the difference, which grows with the data. The price is that a generator can be read only once, so a computation needing two passes has to be written as one.

You'll be able to

Measure peak memory with tracemalloc and compare a list-based and a generator-based computation
Explain why a generator's peak memory doesn't grow with the input
Compute several results (count, total, minimum, maximum) in one pass over an iterator

The idea

How much memory does a computation need? tracemalloc, in Python's standard library, traces the memory Python allocates: start() begins tracing, get_traced_memory() returns (current, peak) in bytes, and stop() ends it. Measuring the peak of two versions of the same computation shows what building a list really costs. Exact byte counts depend on the Python version and the machine, so compare them rather than memorising them.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · A list against a generator

import tracemalloc

def peak_bytes(work):
    tracemalloc.start()
    work()
    current, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return peak

def with_list():
    return sum([n * n for n in range(200_000)])

def with_generator():
    return sum(n * n for n in range(200_000))

list_peak = peak_bytes(with_list)
generator_peak = peak_bytes(with_generator)
print(with_list() == with_generator())
print(list_peak > 1_000_000, generator_peak < 10_000)
print(list_peak > 100 * generator_peak)

The price of laziness: a generator is an iterator, so once something has read it, it's empty. Code that needs two passes, such as an average (total, then count) or a maximum followed by a sum, silently gets wrong answers from a generator. Either keep the values in a list (paying the memory), create a fresh generator for each pass, or compute everything in one pass.

example-2.py · Read once

def fares(rows):
    for row in rows:
        yield float(row["fare_amount"])

rows = [{"fare_amount": "40.8"}, {"fare_amount": "7.2"}, {"fare_amount": "17.5"}]
stream = fares(rows)
print(max(stream))
print(sum(stream))

count = 0
total = 0.0
highest = None
for fare in fares(rows):
    count += 1
    total += fare
    if highest is None or fare > highest:
        highest = fare
print(count, total, highest)

When to use what: a generator when the data is large or endless and one pass is enough; a list when you need len, indexing, sorting or several passes and the data fits comfortably in memory. Mistakes to avoid: calling len() on a generator (TypeError); passing the same generator to two functions; readlines() or read() on a big file when you only loop over it once; and trusting measured bytes as universal: compare versions on the same machine.

Review

tracemalloc measures peak memory: a list's grows with the data, a generator's doesn't.
A generator can be read once; a second consumer gets nothing, often without an error.
Compute several results in one pass, or use a list when you need len, indexing, sorting or several passes.
Next (see Decorators & Context Managers): decorators and context managers, which wrap functions and blocks of code with extra behaviour.

Practise this

Stream the trip file instead of loading it