lesson 112 CLI & Automation · intermediate

Find files with pathlib

Automation starts by finding its input: every CSV in a folder, every export under a tree of subfolders. pathlib lists folders, matches patterns and tells you each file's name, suffix and size.

You'll be able to

List a folder with iterdir() and tell files from folders
Match files with glob("*.csv") and search subfolders with rglob
Sort the results so the script behaves the same on every computer

The idea

A nightly job has to process whatever exports landed in a folder. It can't know the names in advance, so it asks the folder. A Path (see Files & Directories) can do that: folder.iterdir() gives one Path for each entry in the folder, files and subfolders alike.

Each path can describe itself. .name is the last part (invoices-2010-12.csv), .stem is the name without its suffix (invoices-2010-12), .suffix is the final dot-part (.csv), and .is_file() tells a file from a folder. The / operator joins path parts: Path("exports") / "notes.txt" is exports/notes.txt.

The operating system returns folder entries in no particular order, and the order differs between computers. Always wrap the listing in sorted(...) before you print it or act on it.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · What's in the exports folder

from pathlib import Path

exports = Path("exports")
exports.mkdir()
(exports / "invoices-2010-12.csv").write_text("Invoice,Quantity\n489434,12\n")
(exports / "invoices-2011-01.csv").write_text("Invoice,Quantity\n")
(exports / "notes.txt").write_text("sent to finance")
(exports / "archive").mkdir()

for path in sorted(exports.iterdir()):
    if path.is_file():
        print("file:", path.name, "| stem:", path.stem, "| suffix:", path.suffix)
    else:
        print("folder:", path.name)

Usually you want only some files. folder.glob("*.csv") returns the paths in that folder whose names match the pattern: * stands for any run of characters. It looks in that one folder only. folder.rglob("*.csv") searches the folder and every subfolder below it. path.stat().st_size is the file's size in bytes.

example-2.py · Monthly exports spread over year folders

from pathlib import Path

root = Path("shared")
(root / "2010").mkdir(parents=True)
(root / "2011").mkdir()
(root / "2010" / "december.csv").write_text("Invoice,Quantity\n489434,12\n489435,4\n")
(root / "2011" / "january.csv").write_text("Invoice,Quantity\n539993,10\n")
(root / "2011" / "january.csv.bak").write_text("old copy")
(root / "readme.txt").write_text("monthly exports")

print("top level:", [path.name for path in sorted(root.glob("*.csv"))])
total = 0
for path in sorted(root.rglob("*.csv")):
    size = path.stat().st_size
    total += size
    print(path, size)
print("bytes in all exports:", total)

When to use it: whenever a script works on "all the files that…": every CSV in an inbox, every log under a tree, the newest export.

Mistakes to avoid: acting on iterdir() or glob() results without sorting them, so the order changes from one machine to the next; forgetting that iterdir() includes folders; and comparing suffixes without .lower(), so REPORT.CSV (suffix .CSV) is missed. .suffix is only the last dot-part: archive.tar.gz has suffix .gz and stem archive.tar.

Review

iterdir() lists a folder (files and subfolders); glob(pattern) matches in one folder; rglob(pattern) searches below it too.
.name, .stem, .suffix, .is_file() and .stat().st_size describe each path.
Sort listings before using them, and compare suffixes in lower case.
Next: settings that come from the environment instead of the command line.

Practise this