lesson 100 CSV JSON & JSONL · intermediate

JSON Lines: one record per line

A JSON Lines file holds one complete JSON value on each line. You can append a record without rewriting the file, read records one at a time, and a single broken line doesn't spoil the rest.

You'll be able to

Explain why logs and data streams use JSON Lines instead of one big JSON array
Append records with json.dumps(record) + "\n" and read them back line by line
Skip blank lines and report broken ones by line number instead of stopping

The idea

A JSON file holds one value. To add one event to a JSON array you have to read the whole array, append, and write the whole file again, and one broken byte makes the entire file unreadable.

JSON Lines (.jsonl) puts one complete JSON value on each line. Writing a record is out.write(json.dumps(record) + "\n") with mode "a". Reading is for line in file: json.loads(line). Logs, event streams and big dataset exports (like the repository events and complaints in this course) use it because both directions work one record at a time.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Write events, then read them back one by one

import json

events = [
    {"type": "PushEvent", "repo": "tidewater/smartqueue"},
    {"type": "WatchEvent", "repo": "tidewater/smartqueue"},
    {"type": "PushEvent", "repo": "oakline/ledger"},
]
with open("events.jsonl", "w", encoding="utf-8") as out:
    for event in events:
        out.write(json.dumps(event) + "\n")

with open("events.jsonl", encoding="utf-8") as source:
    print(source.read(), end="")
with open("events.jsonl", encoding="utf-8") as source:
    for line in source:
        event = json.loads(line)
        print(event["type"], "in", event["repo"])

A JSONL file is not one JSON document, so json.load on the whole file fails as soon as it reaches the second record.

example-2.py · Reading JSONL as if it were one JSON value

import json

with open("events.jsonl", "w", encoding="utf-8") as out:
    out.write('{"type": "PushEvent"}\n{"type": "WatchEvent"}\n')
with open("events.jsonl", encoding="utf-8") as source:
    events = json.load(source)

Real streams have blank lines (often a trailing one) and, now and then, a line cut off when a writer crashed. Reading one line at a time means you can skip those and keep going, and report which lines were bad so someone can look.

example-3.py · Load what can be loaded, and report the rest

import json

def load_jsonl(path):
    records = []
    bad_lines = []
    with open(path, encoding="utf-8") as source:
        for number, line in enumerate(source, start=1):
            if line.strip() == "":
                continue
            try:
                records.append(json.loads(line))
            except json.JSONDecodeError:
                bad_lines.append(number)
    return (records, bad_lines)

with open("complaints.jsonl", "w", encoding="utf-8") as out:
    out.write('{"complaint_id": "8161310", "state": "MI"}\n')
    out.write("\n")
    out.write('{"complaint_id": "8172405", "sta\n')
    out.write('{"complaint_id": "8172404", "state": "NY"}\n')

records, bad_lines = load_jsonl("complaints.jsonl")
print(len(records), "records, bad lines:", bad_lines)
print(records[-1]["state"])

When to use it: logs, events, exports too big to load at once, and anything that is appended to over time. Mistakes to avoid: writing with indent (one record would span many lines), forgetting the \n after each record, and calling json.load on the whole file.

Review

JSON Lines: one complete JSON value per line, written with json.dumps(record) + "\n" (no indent).
Read it with for line in file: and json.loads(line), skipping blank lines.
Catch json.JSONDecodeError per line to keep going, and record which lines were broken.
Next module: dates and times, exact money, and patterns in text.

Practise this