lesson 221 Data Formats & Analytical Workflows · advanced

Parquet files

CSV stores text, so every reader guesses the types again. Parquet stores typed columns: timestamps stay timestamps, codes stay text, and a reader can load only the columns it needs.

You'll be able to

Round-trip a DataFrame through CSV and through Parquet and compare the dtypes that come back
Read only some columns with read_parquet(path, columns=[...])
Inspect a Parquet file's schema and row count with pyarrow.parquet.ParquetFile

The idea

A CSV file is text. When you read it back, read_csv guesses each column's type again: a timestamp comes back as text, and a zone code such as 007 comes back as the number 7. Parquet is a binary, column-oriented format: each column is stored with its type, compressed, and kept apart from the others. df.to_parquet(path) writes one and pd.read_parquet(path) reads it back with the same dtypes. pandas uses the pyarrow library (Apache Arrow) to do it, which is why this lesson loads pyarrow.

Because columns are stored separately, read_parquet(path, columns=[...]) reads only those columns from disk, which matters when a file has fifty columns and millions of rows.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · What survives a round trip

import pandas as pd

trips = pd.DataFrame({
    "trip_id": [16126, 16127, 16130],
    "pickup_at": pd.to_datetime(["2025-01-15 00:12:47", "2025-01-15 00:18:28", "2025-01-15 00:29:47"]),
    "pickup_zone": ["186", "264", "007"],
    "fare_amount": [40.8, 7.2, 5.1],
})
trips.to_csv("trips.csv", index=False)
trips.to_parquet("trips.parquet", index=False)
from_csv = pd.read_csv("trips.csv")
from_parquet = pd.read_parquet("trips.parquet")
print(from_csv.dtypes.astype(str).tolist(), from_csv["pickup_zone"].tolist())
print(from_parquet.dtypes.astype(str).tolist(), from_parquet["pickup_zone"].tolist())
print(from_parquet.equals(trips))

example-2.py · Reading a CSV as if it were Parquet

import pandas as pd

pd.DataFrame({"trip_id": [16126]}).to_csv("trips.csv", index=False)
pd.read_parquet("trips.csv")

example-3.py · Look inside a file, then read two columns

import pandas as pd
import pyarrow.parquet as pq

trips = pd.DataFrame({
    "trip_id": [16126, 16127, 16128, 16129],
    "pickup_zone_id": [186, 264, 39, 100],
    "payment_type_id": [1, 1, 0, 0],
    "fare_amount": [40.8, 7.2, 17.2, 22.06],
    "tip_amount": [10.7, 2.55, 0.0, 0.0],
})
trips.to_parquet("trips.parquet", index=False)
info = pq.ParquetFile("trips.parquet")
print(info.metadata.num_rows, info.schema_arrow.names)
fares = pd.read_parquet("trips.parquet", columns=["payment_type_id", "fare_amount"])
print(fares.shape, fares.groupby("payment_type_id")["fare_amount"].sum().to_dict())

When to use it: data you will read again, by programs (pipelines, analysis, caches between steps), especially wide or large tables. Keep CSV for exchanging small files with people and spreadsheets. Mistakes to avoid: treating .parquet as text (it is binary); reading every column when you need two; and writing the index you don't want (index=False).

Review

Parquet stores typed, compressed columns: dtypes survive a round trip; CSV makes every reader guess again.
read_parquet(path, columns=[...]) reads only what you need; pyarrow.parquet.ParquetFile shows the schema and row count.
Write with index=False unless the index carries data.
Next: files too big to read at once, processed in chunks.

Practise this

The cache that forgets timestamps