question 454 Reliable Data Pipelines · medium

Which hourly files are left?

A backfill saves the name of the last hourly file it finished. Hour files are named YYYY-MM-DD-HH.jsonl, so name order is time order. Given the folder listing (in whatever order the system returns it) and the checkpoint, work out which files the restart still has to process.

Functionfiles_left(names, checkpoint)

namesfile names in any order; may include other files (README.md, x.jsonl.counts.json) and repeats

checkpoint{"last_done": name}, or {"last_done": None} before anything is finished

Returnsthe .jsonl names that sort after last_done (all of them when it is None), sorted, each once

Rulelast_done might not be in the listing any more (the file was archived); compare by name anyway

Examples

callfiles_left(['2025-03-03-02.jsonl', 'README.md', '2025-03-03-00.jsonl', '2025-03-03-03.jsonl', '2025-03-03-01.jsonl'], {'last_done': '2025-03-03-01.jsonl'})returns['2025-03-03-02.jsonl', '2025-03-03-03.jsonl']

callfiles_left(['2025-03-03-05.jsonl', '2025-03-03-04.jsonl'], {'last_done': '2025-03-03-04.jsonl'})returns['2025-03-03-05.jsonl']

callfiles_left(['2025-03-03-00.jsonl', '2025-03-03-00.jsonl.counts.json'], {'last_done': None})returns['2025-03-03-00.jsonl']