lesson 60 Collection Algorithms · intermediate

Remove duplicates, keep the first

Remember what you have already kept, and skip anything you've seen. One copy of each value or one record per key survives, in first-seen order.

You'll be able to

Remove repeated values while keeping the order they first appeared in
Deduplicate records by one key field, keeping the first record with each key
Choose deliberately between keeping the first and keeping the last

The idea

Exports repeat themselves: a customer orders twice, a payment is retried, a sign-up form is sent again. Deduplicating keeps one copy of each, and the usual rule is *first seen wins*, in the original order.

It is a filter whose test is *have I seen this before?* Keep a second collection of what you've already kept, and check it with not in.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Cities, once each

cities = ["Houston", "New York", "Houston", "Denver", "New York"]
unique = []
for city in cities:
    if city not in unique:
        unique.append(city)
print(unique)

A set (see Dictionaries & Sets) also holds each value once, but it doesn't keep the order you added things in: printing a set of texts can list them in a different order from one run to the next. Use a list when the order matters.

For records, the duplicate test is usually one field, not the whole record. A payment that was retried has a new payment ID but the same order ID. Remember the keys you've seen and keep the first record with each key. A dictionary is a good place for the seen keys, because in on a dictionary checks its keys without walking a list.

example-2.py · First payment attempt per order

def first_attempts(payments):
    seen = {}
    kept = []
    for payment in payments:
        order_id = payment["order_id"]
        if order_id not in seen:
            seen[order_id] = True
            kept.append(payment)
    return kept

payments = [
    {"payment_id": "PAY0000416", "order_id": "O100376", "status": "captured"},
    {"payment_id": "PAY0000417", "order_id": "O100376", "status": "refunded"},
    {"payment_id": "PAY0001023", "order_id": "O100918", "status": "captured"},
    {"payment_id": "PAY0001024", "order_id": "O100918", "status": "refunded"},
    {"payment_id": "PAY0000313", "order_id": "O100282", "status": "captured"},
]
for payment in first_attempts(payments):
    print(payment["payment_id"], payment["order_id"], payment["status"])

Keeping the last record per key is a different rule, with no if at all: store every record under its key, and later records overwrite earlier ones. Pick the rule on purpose. The first sign-up date wants *first*; an order's current status wants *last*. The concept check runs the *last* version.

Review

Deduplicating is a filter whose test is "not seen yet".
Keep first: test not in before storing. Keep last: overwrite under the key every time.
Use the record's key field as the duplicate test, cleaned first when people typed it.
Next (see Comprehensions): comprehensions write the filter and transform patterns in a single expression.

Practise this