runtimeMeasuring memory: getsizeof, tracemalloc snapshots and __slots__
Memory problems are found by measuring, not guessing. sys.getsizeof tells you about one object, tracemalloc tells you which lines allocated memory, and __slots__ shrinks objects you create by the million.
- Explain why
sys.getsizeofof a container excludes the objects it holds - Compare two tracemalloc snapshots to find the lines that allocated the most
- Use
__slots__for many small objects, and know what it gives up
sys.getsizeof(obj) returns the bytes one object occupies. For a container it is shallow: a list's size is its header plus one slot (a reference) per item. The items themselves live elsewhere and are not counted, so a list of long strings and a list of short strings with the same length report the same size. Exact numbers depend on the Python version and the machine (references are 4 bytes on a 32-bit build like this browser's, 8 on most desktops), so compare sizes rather than memorising them.
Run each example to see what it prints. Running examples doesn't count toward progress.
import sys short_names = ["Ana", "Ben", "Chen"] * 1000 long_names = ["Alexandria Konstantinopoulou-Whitfield"] * 3000 print(len(short_names) == len(long_names)) print(sys.getsizeof(short_names) == sys.getsizeof(long_names)) print(sys.getsizeof("Ana") < sys.getsizeof("Alexandria Konstantinopoulou-Whitfield"))
To find where memory goes, tracemalloc records every allocation together with the line that made it. Take a snapshot before and after some work; after.compare_to(before, "lineno") lists the lines whose memory grew, biggest first. Each entry has a size_diff (bytes) and a traceback whose first frame has the lineno.
import tracemalloc tracemalloc.start() before = tracemalloc.take_snapshot() trips = [{"trip_id": i, "fare": 12.5} for i in range(20000)] labels = [f"trip-{i}" for i in range(20000)] after = tracemalloc.take_snapshot() tracemalloc.stop() for stat in after.compare_to(before, "lineno")[:2]: frame = stat.traceback[0] print("line", frame.lineno, "grew by more than 500 KiB:", stat.size_diff > 500 * 1024)
A common fix for *millions of small objects* is __slots__. Normally every instance carries its own attribute dictionary (__dict__). Listing the attribute names in __slots__ gives each instance fixed slots instead, which saves memory per object, at a price: the class can't get new attributes at run time.
import tracemalloc class Trip: def __init__(self, trip_id, fare): self.trip_id = trip_id self.fare = fare class SlimTrip: __slots__ = ("trip_id", "fare") def __init__(self, trip_id, fare): self.trip_id = trip_id self.fare = fare def traced_bytes(make): tracemalloc.start() kept = make() current, peak = tracemalloc.get_traced_memory() tracemalloc.stop() return current plain = traced_bytes(lambda: [Trip(i, 12.5) for i in range(10000)]) slim = traced_bytes(lambda: [SlimTrip(i, 12.5) for i in range(10000)]) print(slim < plain, plain / slim > 1.2) print(hasattr(Trip(1, 9.5), "__dict__"), hasattr(SlimTrip(1, 9.5), "__dict__"))
class SlimTrip: __slots__ = ("trip_id", "fare") def __init__(self, trip_id, fare): self.trip_id = trip_id self.fare = fare trip = SlimTrip(16126, 40.8) trip.tip = 10.7
When to use which: getsizeof for a quick look at one object (remember it is shallow); tracemalloc snapshots to find which lines allocate (and get_traced_memory() for current and peak totals); __slots__ only for classes with very many instances. Mistakes to avoid: believing a container's getsizeof is the memory of everything in it; measuring once on tiny data; and adding __slots__ everywhere, which makes classes rigid for no measurable gain.
sys.getsizeofis shallow: a container's own size, not its contents.- tracemalloc snapshots +
compare_to(…, "lineno")show which lines grew;get_traced_memory()gives current and peak. __slots__removes the per-object__dict__: less memory for many instances, but no new attributes.- Next: what CPython does when it runs your code, bytecode and the GIL.