Kevin Kevin - 3 months ago 19
Python Question

Python: Dedup a dictionary

Good afternoon,

I reading in a pcap and am basically trying to get a dedup'd list of BSSID's & ESSID's. I am still getting duplicates with this code and cannot for the life of me figure out what I am doing wrong:

if not (t[0] in ssid_pairs and ssid_pairs[t[0]] == t[1]):
ssid_pairs[t[0]] = t[1]
of.write(t[0] + ',' + t[1] + ((',' + f + '\n') if verbose else '\n'))


ssid_pairs is a dictionary, t[0] is the bssid & t[1] is the essid. An example of the dictionary is:

{'FF:FF:FF:FF:FF:FF':'MyWIFI',...}


I am still seeing multiple instances of the same key->value pair being written to the file. I put some debugging print statements in and sometimes it will recognize a duplicate, sometimes it will not. This is from a parse pcap with scapy.

Thanks for any help.

*** EDIT: Thanks everyone, I am not really solving my problem the right way with a dictionary. Time to think this through a bit clearer...

Answer

Let's say you get:

t[0] = 'foo'
t[1] = 'bar'

Then we hit your code above:

if not (t[0] in ssid_pairs and ssid_pairs[t[0]] == t[1]):
    ssid_pairs[t[0]] = t[1]
    of.write(t[0] + ',' + t[1] + ((',' + f + '\n') if verbose else '\n'))

The condition passes (because t[0] is not in ssid_pairs), so we set:

ssid_pairs[t[0]] = t[1]

Which gives us:

ssid_pairs = {
  'foo': 'bar',
}

In our next iteration of the loop, we read:

t[0] = 'foo'
t[1] = 'gizmo'

Your condition passes (because ssid_pairs[t[0]] != t[1]), so we set:

ssid_pairs[t[0]] = t[1]

Which gives us:

ssid_pairs = {
  'foo': 'gizmo',
}

Then we read the same data we encountered in our first iteration:

t[0] = 'foo'
t[1] = 'bar'

But because we just ssid_pairs['foo'] = 'gizmo', your condition will pass again, and you will once again write out the same data.

If you are trying to get a list of unique pairs, then maybe create a set of (essid,bssid) tuples:

seen_pairs = set()
...
if not ((t[0],t(1)) in seen_pairs):
    seen_pairs.add((t[0], t[1]))
    of.write(t[0] + ',' + t[1] + ((',' + f + '\n') if verbose else '\n'))