question 457 Reliable Data Pipelines · medium

Flatten archive events into rows

Archive events are nested: the account is under actor.login, the repository under repo.name, and a push's commit count under payload.size. Turn a list of raw events into flat rows a table can hold, and report the events that can't be flattened instead of guessing.

Functionflatten(raw_events)

Row{"id", "type", "actor", "repo", "created_at", "commits"}; commits is payload["size"] for a PushEvent and None for every other type

Probleman event with no actor.login or no repo.name (missing, None or empty) is not turned into a row; report "<id>: missing actor" or "<id>: missing repo" (actor is checked first; one problem per event)

Returns(rows, problems) in input order; the input events are not changed

Examples

callflatten([{'id': '48000000581', 'type': 'PushEvent', 'actor': {'login': 'ml-koi418'}, 'repo': {'name': 'tidewater/smartqueue'}, 'payload': {'size': 2}, 'created_at': '2025-03-03T00:02:37Z'}, {'id': '48000007504', 'type': 'WatchEvent', 'actor': {'login': 'stack_yak150'}, 'repo': {'name': 'lumen-labs/pyauth'}, 'payload': {'action': 'started'}, 'created_at': '2025-03-03T00:09:20Z'}, {'id': '48000002968', 'type': 'ForkEvent', 'actor': None, 'repo': {'name': 'lumen-labs/tinykit'}, 'payload': {}, 'created_at': '2025-03-03T00:19:28Z'}])returns([{'id': '48000000581', 'type': 'PushEvent', 'actor': 'ml-koi418', 'repo': 'tidewater/smartqueue', 'created_at': '2025-03-03T00:02:37Z', 'commits': 2}, {'id': '48000007504', 'type': 'WatchEvent', 'actor': 'stack_yak150', 'repo': 'lumen-labs/pyauth', 'created_at': '2025-03-03T00:09:20Z', 'commits': None}], ['48000002968: missing actor'])

callflatten([{'id': 'e1', 'type': 'IssuesEvent', 'actor': {'login': 'dev_bee483'}, 'repo': {}, 'payload': {}, 'created_at': 't'}])returns([], ['e1: missing repo'])

callflatten([{'id': 'e2', 'type': 'PushEvent', 'actor': {'login': 'a'}, 'repo': {'name': 'o/r'}, 'payload': {'size': 3}, 'created_at': 't2'}])returns([{'id': 'e2', 'type': 'PushEvent', 'actor': 'a', 'repo': 'o/r', 'created_at': 't2', 'commits': 3}], [])