lesson 237 Observability & Operations · advanced

Count and time what the program does

Logs answer "what happened to this request?". Metrics answer "how is the whole system doing?": how many requests, how many failed, how long they took. Durations are summarised with percentiles, because an average hides the slow requests users actually notice.

You'll be able to

Count outcomes with counters and measure durations with a monotonic clock passed in as a function
Compute a nearest-rank percentile and explain why p95 tells you more than the mean
Group measurements per operation or endpoint and report them

The idea

A metric is a number measured over time: requests served, errors, items in a queue, how long each call took. Two kinds cover most needs. A counter only goes up (requests, errors, rejected lines). A distribution collects measurements (durations, sizes) and is summarised.

The summary matters. Response times are skewed: most requests are fast and a few are very slow. The mean mixes them into a number that describes nobody. A percentile says *p% of measurements were at or below this*: p50 (the median) is the typical case, p95 is what the slowest one in twenty users waits. The simplest definition is nearest rank: sort the values, and p is the value at position ceil(p / 100 × n) (counting from 1).

Durations come from a clock. time.perf_counter() is monotonic (it never jumps when the wall clock is corrected), so subtracting two readings gives an elapsed time. Passing the clock in as a function (the Advanced OOP & Design module's dependency injection) lets tests and these examples use exact, repeatable times.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Why p95, not just the mean

import math
import statistics

def percentile(values, p):
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered))
    return ordered[max(rank, 1) - 1]

api_ms = [112, 98, 105, 120, 101, 97, 110, 2350, 103, 99, 108, 115, 102, 96, 104, 1890, 107, 100, 111, 109]

print("requests", len(api_ms))
print("mean", statistics.mean(api_ms))
print("p50", percentile(api_ms, 50), "p95", percentile(api_ms, 95), "max", max(api_ms))

Summaries need data. With no measurements at all, a mean doesn't exist, and statistics says so instead of returning 0:

example-2.py · The mean of nothing

import statistics

timings = []
print(statistics.mean(timings))

In a program, metrics live in one small object: counters in a Counter, measurements in lists per name, and a helper that times a call with the injected clock. A try/finally records the duration even when the call fails, because slow failures are exactly what you want to see.

example-3.py · A metrics object with a timer

import json
import time
from collections import Counter, defaultdict

class Metrics:
    def __init__(self, clock=time.perf_counter):
        self.clock = clock
        self.counts = Counter()
        self.timings = defaultdict(list)

    def inc(self, name):
        self.counts[name] += 1

    def timed(self, name, func, *args):
        start = self.clock()
        try:
            return func(*args)
        finally:
            self.timings[name].append(self.clock() - start)

readings = iter([0.0, 0.25, 1.0, 1.5, 2.0, 2.75])
metrics = Metrics(clock=readings.__next__)

for text in ['{"type": "PushEvent"}', '{"type": "WatchEv', '{"type": "ForkEvent"}']:
    try:
        metrics.timed("parse", json.loads, text)
        metrics.inc("parsed")
    except json.JSONDecodeError:
        metrics.inc("rejected")

print(dict(metrics.counts))
print(metrics.timings["parse"])

When to use it: for every operation whose speed or failure rate matters: requests, queries, pipeline steps. Mistakes to avoid: reporting only the mean; time.time() for durations (it can jump); timing only successful calls; and sorting the caller's list in place to compute a percentile, which reorders their data.

Review

Counters count events; distributions collect measurements and are summarised.
Report count, p50, p95 and max: the mean hides the slow tail. Nearest rank: position ceil(p/100 × n) of the sorted values.
Measure with a monotonic clock passed in as a function, and record durations of failures too.
Next: one request crosses many steps. Tracing follows it with a trace id and spans.

Practise this