lesson 219 Visualization & EDA · advanced

Distributions and outliers

Before trusting an average, look at the spread: a histogram counts values per bin, quantiles mark the middle half, and the 1.5 × IQR rule flags values far outside it.

You'll be able to

Count values per bin with np.histogram or ax.hist, choosing bin edges on purpose
Summarise a column with describe and quantile, and explain the median and quartiles
Flag outliers with the 1.5 × IQR rule and show them with a box plot

The idea

Two columns can share a mean and look nothing alike. Exploratory data analysis starts by looking at each column's distribution. A histogram splits the range into bins (edges such as 60, 90, 120, 150 make three bins: 60–90, 90–120, 120–150, each including its left edge) and counts the values in each. np.histogram(values, bins=edges) returns the counts and the edges; ax.hist draws the same bins and returns them too.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Film runtimes in 30-minute bins

import matplotlib.pyplot as plt
import numpy as np

runtimes = [107, 107, 87, 108, 133, 117, 79, 124, 69, 95, 126, 99, 95, 75, 134, 104]   # Runtime Films
counts, edges = np.histogram(runtimes, bins=[60, 90, 120, 150])
print(counts.tolist(), edges.tolist())
fig, ax = plt.subplots()
drawn, _, bars = ax.hist(runtimes, bins=[60, 90, 120, 150])
ax.set_xlabel("Runtime (min)")
print(drawn.tolist(), len(bars))
plt.show()

example-2.py · Bin edges out of order

import numpy as np

counts, edges = np.histogram([107, 87, 133], bins=[60, 120, 90, 150])

Numbers summarise the same picture. describe() gives the count, mean, standard deviation, minimum, the quartiles (25%, 50% = median, 75%) and maximum. The middle half of the data lies between the first and third quartile; that width is the IQR (interquartile range). A common rule calls a value an outlier when it is more than 1.5 × IQR below the first quartile or above the third. It is a rule for *looking closer*, not for deleting: the outlier might be the most important value in the data.

example-3.py · Which delays are extreme?

import matplotlib.pyplot as plt
import pandas as pd

delays = pd.Series([-4, 12, -25, -3, 42, -22, -23, 18, 54, 167, 5, -22, -49, 0, -2, 0])   # arrival delays (min), D13
print(delays.describe().round(1).to_dict())
q1, q3 = delays.quantile(0.25), delays.quantile(0.75)
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print(q1, q3, low, high)
print(delays[(delays < low) | (delays > high)].tolist())
fig, ax = plt.subplots()
ax.boxplot(delays)
plt.show()

Mistakes to avoid: trusting the mean of skewed data; histograms with too few or too many bins (try two or three widths); bins that leave the maximum out (np.histogram includes the last edge in the last bin, but a hand-made rule might not); and deleting outliers without a reason that is about the data, not the chart.

Review

A histogram counts values per bin; choose the edges on purpose (np.histogram, ax.hist).
describe and quantile summarise the spread; the median resists extremes.
1.5 × IQR beyond the quartiles flags outliers to investigate; a box plot shows them.
Next: turn a chart into a clear statement.

Practise this