lesson 209 pandas Foundations · advanced

Missing values

An empty cell becomes NaN. isna finds the gaps, dropna and fillna deal with them, and sum and mean quietly skip them, which is what you want only if you know it happens.

You'll be able to

Count missing values per column with isna().sum() and test single cells with pd.isna
Remove rows with dropna(subset=...) and fill gaps with fillna, on purpose
Explain that mean, sum and count skip NaN, and why filling with 0 changes an average

The idea

Real tables have holes: an order with no promo code, a review with no title, a shipment that hasn't been delivered yet. read_csv turns empty cells into missing values, shown as NaN (in number columns, and also in text columns in pandas 3). A missing value is not 0 and not an empty string: it means *unknown*.

df.isna() is a True/False table of where the gaps are, so df.isna().sum() counts them per column. notna() is the opposite. For one value, use pd.isna(value): == can't detect NaN, because NaN is not equal to anything, itself included.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Where are the gaps?

import io
import pandas as pd

csv_text = """order_id,promo_code,shipping_fee,delivered_at
O100282,WELCOME10,5.99,2024-01-18 07:35:32
O100376,FREESHIP,,2024-01-24 20:02:08
O100918,,5.99,
O101219,,0,2024-03-19 01:26:45
"""
orders = pd.read_csv(io.StringIO(csv_text))
print(orders.isna().sum())
print(orders["promo_code"].notna().sum(), "orders used a code")
print(pd.isna(orders.loc[2, "delivered_at"]))

example-2.py · Converting a missing rating

import pandas as pd

reviews = pd.DataFrame({"rating": [5, None, 4]})
print(reviews["rating"].tolist())
stars = int(reviews["rating"].iloc[1])

Three ways to deal with gaps, chosen per column: drop the rows (dropna(subset=["col"])) when the row is useless without that value; fill (fillna(value)) when there is a correct default, such as "none" for no promo code; or leave them, because sum, mean and count already skip NaN. Filling with 0 is a decision, not a cleanup: it changes every average.

example-3.py · Average rating: skip, fill, or drop?

import pandas as pd

reviews = pd.DataFrame({
    "review_id": ["R000103", "R000140", "R000188", "R000201"],
    "rating": [5, 2, None, 4],
    "title": ["Great quality", None, "Works", None],
})
print(reviews["rating"].mean(), reviews["rating"].count(), len(reviews))
print(reviews["rating"].fillna(0).mean())
titled = reviews.dropna(subset=["title"])
print(titled["review_id"].tolist())
print(reviews["title"].fillna("(no title)").tolist())

When to use it: check isna().sum() on every new file before computing anything. Mistakes to avoid: value == None or value == float("nan") (always False for NaN; use pd.isna), filling numbers with 0 without a reason, and dropping rows for a gap in a column the question doesn't use (dropna() with no subset drops a row for a gap anywhere).

Review

Empty cells become NaN; isna().sum() counts gaps per column; pd.isna(x) tests one value.
dropna(subset=...) removes rows, fillna(value) fills gaps; both return new data.
sum, mean and count skip NaN; filling with 0 changes the answer.
Next: each column's dtype, and converting text that should be numbers.

Practise this