lesson 178 Memory & Python Internals · advanced

Garbage collection: cycles and weak references

Two objects that refer to each other keep each other's reference count above zero forever. CPython's cycle collector finds such islands; weak references avoid creating them.

You'll be able to

Explain why a reference cycle is not freed by reference counting
Run the cycle collector with gc.collect() and observe objects with weakref.ref
Use weak references for back-links, listeners and caches

The idea

Reference counting frees an object when nothing refers to it. But if an order refers to its invoice and the invoice refers back to the order, each keeps the other's count at 1 even after the program has dropped both. That island of unreachable objects is a reference cycle. CPython has a second mechanism for it, the cycle collector in the gc module: every so often it looks for groups of objects that are only reachable from each other and frees them. gc.collect() runs it now and returns how many unreachable objects it found.

To watch an object without keeping it alive, use a weak reference: weakref.ref(obj) returns a callable that gives back the object while it exists and None once it has been freed. The example pauses the automatic collector (gc.disable()) so you can see exactly when the cycle is freed.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · A cycle survives until the collector runs

import gc
import weakref

class Node:
    def __init__(self, name):
        self.name = name
        self.partner = None

gc.disable()
order = Node("order O100282")
invoice = Node("invoice INV-881")
order.partner = invoice
invoice.partner = order
watch = weakref.ref(order)
del order, invoice
print(watch() is None)
print(gc.collect() >= 2, watch() is None)
gc.enable()

Waiting for the collector means memory is held longer and cleanup happens at an unpredictable time. Better is not to make the cycle: when a child needs to reach its parent, store a weak reference to the parent. The parent keeps its children alive; the children don't keep the parent alive.

example-2.py · Children that don't keep their parent alive

import weakref

class Category:
    def __init__(self, name, parent=None):
        self.name = name
        self.children = []
        self.parent_ref = None
        if parent is not None:
            self.parent_ref = weakref.ref(parent)
            parent.children.append(self)

    def path(self):
        parent = None
        if self.parent_ref is not None:
            parent = self.parent_ref()
        if parent is None:
            return self.name
        return parent.path() + " > " + self.name

home = Category("Home")
kitchen = Category("Kitchen", home)
print(kitchen.path())
watch = weakref.ref(home)
del home
print(watch() is None, kitchen.path())

The weakref module also offers WeakValueDictionary (a dictionary whose values disappear when nothing else uses them, ideal for caches) and weakref.finalize(obj, callback) (run a cleanup when an object is freed). Not every object can be weakly referenced: instances of your own classes can, but plain int, str, list and dict objects can't (TypeError: cannot create weak reference).

When to use it: back-links (child → parent, item → owner), listener lists, and caches that shouldn't keep objects alive. Mistakes to avoid: relying on __del__ or the collector for important cleanup (close files with with instead); calling gc.collect() routinely to *fix* memory (find and break the cycle instead); and forgetting that a weak reference can return None at any time.

Review

Objects that refer to each other form a cycle: reference counting can't free it, the cycle collector (gc) can.
gc.collect() runs the collector now; normally it runs on its own, at moments you don't control.
weakref.ref, WeakValueDictionary and finalize observe objects without keeping them alive.
Next: measuring how much memory your objects take, and where it was allocated.

Practise this