runtimeProcesses: parallel work, separate memory
Each process has its own interpreter, its own memory and its own GIL, so CPU-heavy Python really runs on several cores at once. The price: everything sent to a process and back is copied by pickling.
Read and diagnose: this can't run in the browser, so you'll read real code and output and answer questions about it.
- Explain why processes, not threads, speed up CPU-bound Python on standard CPython
- Know what crosses a process boundary: pickled arguments and results; functions by name
- Split work into chunks, map a top-level function over them, and combine the partial results
Threads share one interpreter and its GIL, so pure-Python computation in threads runs one thread at a time. A process is a whole separate Python interpreter, with its own memory and its own GIL, so eight processes can compute on eight cores at once. concurrent.futures.ProcessPoolExecutor gives processes the same submit/map interface as the thread pool (“Executors and futures”).
Processes don't share memory. Arguments travel to the worker and results travel back as bytes made by pickle, Python's object serializer. That has consequences you can see right here, because pickling works in this page even though starting processes doesn't.
Run each example to see what it prints. Running examples doesn't count toward progress.
import importlib import pickle with open("fares.py", "w", encoding="utf-8") as f: f.write("def zone_total(rows):\n return sum(row['fare'] for row in rows)\n") importlib.invalidate_caches() from fares import zone_total job = (zone_total, [{"fare": 40.8}, {"fare": 7.2}]) data = pickle.dumps(job) func, rows = pickle.loads(data) print(func is zone_total, func(rows), rows is job[1]) try: pickle.dumps(lambda rows: len(rows)) except pickle.PicklingError: print("PicklingError")
from concurrent.futures import ProcessPoolExecutor with ProcessPoolExecutor(max_workers=2) as pool: print(pool.submit(len, "SPO-11681").result())
Process pools work best with a few big jobs: each job pays for pickling its data and its result, and starting the worker processes takes time. So split the input into one chunk per worker, map a top-level function over the chunks, and combine the partial results in the parent. The example runs that shape with the built-in map; on a computer you replace map with pool.map.
from collections import Counter def count_zones(lines): counts = Counter() for line in lines: counts[line.split(",")[1]] += 1 return counts def chunked(items, size): return [items[i:i + size] for i in range(0, len(items), size)] lines = [f"{16126 + i},{(i * 7) % 5}" for i in range(20)] # On a computer, under the __main__ guard: with ProcessPoolExecutor() as pool: partials = list(pool.map(count_zones, chunked(lines, 6))) partials = list(map(count_zones, chunked(lines, 6))) total = Counter() for part in partials: total += part print(len(partials), dict(sorted(total.items())))
When to use processes: CPU-heavy pure-Python work (parsing, simulation, image or text processing) that can be split into independent chunks. When not to: small jobs (start-up and pickling cost more than the work), jobs needing large shared data (it is copied to every worker), and waiting-bound work (threads or asyncio are lighter). Mistakes to avoid: lambdas or nested functions as the worker; expecting a worker's changes to globals to be visible in the parent; and forgetting the __main__ guard.
- Processes have separate memory and separate GILs: CPU-bound Python runs in parallel.
- Arguments and results are pickled copies; worker functions must be top-level; never unpickle untrusted data.
- Few big chunks, a top-level worker, combine partial results; guard the entry point with
__main__. - Next: measured numbers that show when threads or processes actually pay off.