lesson 184 Profiling & Optimization · advanced

Efficient I/O: stream, open once, write in batches

Files are slow and can be huge. Reading line by line keeps memory flat whatever the size, and opening a file once and writing in batches avoids paying the cost of the disk again and again.

You'll be able to

Compare the peak memory of reading a whole file with streaming it, using tracemalloc
Keep only what is needed while streaming (counts, a deque of recent lines)
Open each output file once and write in batches

The idea

f.read() and f.readlines() put the whole file in memory, plus a list of every line. Iterating for line in f (see Files & Directories) reads a buffer-full at a time and hands out one line at a time, so memory stays at about the size of the buffer, whether the file is 1 MB or 10 GB. The example writes a 50,000-line pickup file and measures both ways with tracemalloc (“Measuring memory: getsizeof, tracemalloc snapshots and __slots__”).

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · The same count, two memory profiles

import tracemalloc

with open("pickups.csv", "w", encoding="utf-8") as f:
    f.write("trip_id,zone\n")
    for i in range(50000):
        f.write(f"{16126 + i},{i % 263}\n")

def peak(work):
    tracemalloc.start()
    result = work()
    current, top = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return result, top

def count_whole():
    with open("pickups.csv", encoding="utf-8") as f:
        lines = f.read().splitlines()
    return sum(1 for line in lines[1:] if line.endswith(",132"))

def count_streaming():
    count = 0
    with open("pickups.csv", encoding="utf-8") as f:
        next(f)
        for line in f:
            if line.rstrip("\n").endswith(",132"):
                count += 1
    return count

(a, whole), (b, stream) = peak(count_whole), peak(count_streaming)
print(a, b, whole > 10 * stream)

Writing has its own trap. Opening a file is expensive (the operating system checks permissions, finds the file, sets up buffers), and every open in a loop pays it again, while every write on an open file goes into a buffer that is flushed in large blocks. So: open once, write many times, or build a batch and write it with one call (f.write("".join(lines)) or f.writelines(lines)). The example counts the calls with a small wrapper.

example-2.py · Opening per line against opening once

opens = {"count": 0}

def counted_open(path, mode):
    opens["count"] += 1
    return open(path, mode, encoding="utf-8")

events = [f"order {n} shipped\n" for n in range(1000)]

for event in events:
    with counted_open("per_line.log", "a") as log:
        log.write(event)
per_line = opens["count"]

opens["count"] = 0
with counted_open("once.log", "w") as log:
    log.writelines(events)

with open("per_line.log", encoding="utf-8") as a, open("once.log", encoding="utf-8") as b:
    print(a.read() == b.read())
print(per_line, opens["count"])

When to use it: stream every file that could grow (logs, exports, sensor data); keep only aggregates or a bounded window (collections.deque(maxlen=n), the Collections & Itertools module) while reading. Keep output files open for the whole job, or collect a batch and write it at once. Mistakes to avoid: read()/readlines() on files of unknown size; building a list of all rows when you only need a total; opening a file inside a loop; and forgetting with, which leaves files open (and unflushed) after an error.

Review

for line in f streams a file in buffer-sized pieces: memory stays flat at any file size.
Keep only what you need while streaming: totals, or deque(maxlen=n) for the last n items.
Open each output once (or write a batch at once) and close everything with with or finally.
Next: the biggest wins of all, changing the algorithm, and proving the result is unchanged.

Practise this

Copy the matching lines of a huge log