Atirag Atirag - 1 month ago 6
Python Question

Python: Loop through list of lists matching a value and changing it

What will be the fastest way to do this? I have a list of lists and I have to loop through it looking for a particular value on each of the sublists. Once the value is found I have to append a value to another separate list and then change the value that I found. I started doing this

for x in f
if 'nan' in x:
ef.append(1.0)
else:
ef.append(0.0)


f is the list of lists so I loop through each list x and check if there's a nan value among the elements of the sublist. If there's at least one nan value among the elements of the sublist x I append 1.0 to ef but if there isn't I append 0.0. Now I need to do this but at the same time I have to change all the nan values on each sublist to 1.0. So I want to do something like

for x in f:
if 'nan' in x:
ef.append(1.0)
x[position of nans] = 1.0
else:
ef.append(0.0)


Is there a way to do this within this loop or must I loop again? I'm dealing with very large quantities of data so I wanted to keep it as fast as possible.

Answer

The complexity for something like this is linear, you couldn't do better, because you need to inspect every element at least once. Assuming you want to append 1 or 0 in ef list only once per sublist, something like this will work:

f = [[1, 'nan'], [2, 'nan', 'nan']]
ef = []

for x in f:
    found = False
    for pos, e in enumerate(x):
        if e == 'nan':
            x[pos] = 1
            found = True
    ef.append(1 if found else 0)

print f, ef

The live example on ideone here.

Timing on my laptop (includes building the list):

$ time python test.py 
numElements = 15000000

real    0m1.908s
user    0m1.879s
sys 0m0.028s
Comments