ShriShinde ShriShinde - 28 days ago 12
Python Question

How to efficiently filter data from list of dictionaries

I have data in

tuple_list
and
dictionaries_list
and in both the lists there can be thousands of records.
I want to filter data from
dictionaries_list
based on records from
tuple_list
. Currently I wrote following code but it takes lot of time as for each element from list of tuples it has to iterate entire list of dictionaries.

# tuple_list can be like [(('s', 45), ('t', 30)), (('s', 5), ('t', 3))]
for target_tuple in tuple_list:
# target_tuple can have data like (('s', 45), ('t', 30))
# dictionaries_list can have data like [{'a': 5, 's': 45, 't': 10}, {}, {}]
if some_parameter == 'something':
m1_dicts = [d for d in dictionaries_list if d['s'] == target_tuple[0][1]]
else:
m1_dicts = [d for d in dictionaries_list if d['t'] == target_tuple[0][1]]


Please suggest some way to improve this.

Answer

From what i understand, you wanna keep the dictionary if any of the tuple_list found match within the dictionary. It did improve a bit on runtime because it only run through tuple_list once, but might not the best optimize yet as it still need to iterate through list of dictionaries

# taking small dataset as example
tuple_list=[(('s', 45), ('t', 30)), (('s', 5), ('t', 6))]
dictionaries_list = [{'a': 5, 's': 45, 't': 10}, {'s':5,'t':6}, {'a':2,'s':22,'t':30}]

tuple_dict = {}
tuple_dict['s'] = [i[0][1] for i in tuple_list]
tuple_dict['t'] = [i[1][1] for i in tuple_list]
if some_parameter == 'something':
    # check for 's'
    m1_dicts = [d for d in dictionaries_list if d['s'] in tuple_dict['s']]
    # m1_dicts = [{'a': 5, 's': 45, 't': 10}, {'s':5,'t':6}]
else:
    # check for 't'
    m1_dicts = [d for d in dictionaries_list if d['t'] in tuple_dict['t']]
    # m1_dicts = [{'s':5,'t':6}, {'a':2,'s':22,'t':30}]

Instead of some_parameter = 'something', suggest to use some_patameter as the key to check for

some_parameter = 's' # check for 's'
m1_dicts = [d for d in dictionaries_list if d[some_parameter] in tuple_dict[some_parameter]]