question 459 Reliable Data Pipelines · hard

A loader that survives a crash

Put the module together: load a JSON Lines event file into SQLite so that bad lines are counted rather than fatal, a rerun never duplicates a row, and a crash part way through resumes from a checkpoint. Write load_file(path, conn, ckpt_path).

Functionload_file(path, conn, ckpt_path); the table events (event_id TEXT PRIMARY KEY, type TEXT) already exists

Per lineskip blank lines; a line that isn't a JSON object with id and type is counted as rejected; otherwise insert it, ignoring an ID that is already stored

Checkpoint{"line": n} = lines finished (blank and rejected lines count as finished); after every 2 lines, conn.commit() and then save the checkpoint atomically; at the end commit, then delete the checkpoint

Resumewith a checkpoint, start after line n

Returns(loaded, rejected) counted by this call (an ID already stored still counts as loaded)

Crashan error from the database is not caught: it stops the run