lesson 95 Files & Directories · intermediate

Text, bytes and encodings

On disk a file is bytes. An encoding turns characters into bytes and back; the wrong one either fails with UnicodeDecodeError or quietly garbles accented names.

You'll be able to

Explain why a character can take more than one byte in UTF-8, and measure text in bytes
Recognise UnicodeDecodeError and garbled text (CafÃ©) as encoding mismatches
Read a file as bytes and decode it, falling back to another encoding when UTF-8 fails

The idea

A file stores bytes, numbers from 0 to 255. Text is made of characters, and there are far more than 256 of them. An encoding is the rule that turns characters into bytes and back.

UTF-8 uses one byte for plain English letters and digits, and two to four bytes for everything else (ë, ü, €, 東, emoji). text.encode("utf-8") gives you the bytes, and data.decode("utf-8") turns bytes back into text. The encoding= you pass to open does exactly this for you on every read and write.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Characters versus bytes

name = "Zoë Müller"
data = name.encode("utf-8")
print(data)
print(len(name), "characters,", len(data), "bytes")
print(data.decode("utf-8") == name)
print(data.decode("latin-1"))

The other way round fails loudly. Older Windows programs often save text as cp1252, where é is the single byte \xe9. In UTF-8 that byte can't start a character, so decoding stops with UnicodeDecodeError.

example-2.py · cp1252 bytes read as UTF-8

data = "Café crème".encode("cp1252")
print(data)
print(data.decode("utf-8"))

When files come from many places, you can't always know the encoding in advance. A practical rule: try UTF-8 first (bytes from another encoding almost never happen to be valid UTF-8), and fall back to the likely legacy encoding only when it fails. Opening with mode "rb" reads the raw bytes, so you can decide.

example-3.py · Read a menu saved by an old program

def read_text_any(path):
    with open(path, "rb") as source:
        data = source.read()
    try:
        return data.decode("utf-8")
    except UnicodeDecodeError:
        return data.decode("cp1252")

with open("menu_old.txt", "w", encoding="cp1252") as out:
    out.write("Crème brûlée,6.50\n")
with open("menu_new.txt", "w", encoding="utf-8") as out:
    out.write("Crème brûlée,6.50\n")
print(read_text_any("menu_old.txt") == read_text_any("menu_new.txt"))
print(read_text_any("menu_old.txt"), end="")

When to use it: whenever text crosses a boundary: files, network, databases. Measure bytes when a limit is in bytes (database columns, message sizes). Mistakes to avoid: leaving out encoding=, assuming one character is one byte, and "fixing" a UnicodeDecodeError with errors="ignore", which silently deletes characters.

Review

Files hold bytes; an encoding maps characters to bytes. UTF-8 uses 1–4 bytes per character.
encode turns text into bytes and decode turns bytes into text; measure in bytes when a limit is in bytes.
The wrong encoding raises UnicodeDecodeError or quietly garbles text; read bytes with "rb" to decide yourself.
Next module: files with structure. CSV rows, JSON documents and JSON Lines.

Practise this

Which supplier files aren't UTF-8?