Brenden Brenden - 4 months ago 20
Python Question

Remove duplicate dict in list in Python

I have a list of dicts, and I'd like to remove the dicts with identical key and value pairs.

For this list:

[{'a': 123}, {'b': 123}, {'a': 123}]


I'd like to return this:
[{'a': 123}, {'b': 123}]


Another example:

For this list:
[{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}, {'a': 123, 'b': 1234}]


I'd like to return this:
[{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}]

Answer

Try this:

[dict(t) for t in set([tuple(d.items()) for d in l])]

The strategy is to convert the list of dictionaries to a list of tuples where the tuples contain the items of the dictionary. Since the tuples can be hashed, you can remove duplicates using set and, after that, re-create the dictionaries from tuples with dict.

where:

  • l is the original list
  • d is one of the dictionaries in the list
  • t is one of the tuples created from a dictionary

Edit: If you want to preserve ordering, the one-liner above won't work since set won't do that. However, with a few lines of code, you can also do that:

l = [{'a': 123, 'b': 1234},
        {'a': 3222, 'b': 1234},
        {'a': 123, 'b': 1234}]

seen = set()
new_l = []
for d in l:
    t = tuple(d.items())
    if t not in seen:
        seen.add(t)
        new_l.append(d)

print new_l

Example output:

[{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}]

Note: As pointed out by @alexis it might happen that two dictionaries with the same keys and values, don't result in the same tuple. That could happen if they go through a different adding/removing keys history. If that's the case for your problem, then consider sorting d.items() as he suggests.

Comments