mac mac - 1 month ago 18
Python Question

Weird lambda behaviour in loops

I stumbled upon a behaviour in python that I have a hard time understanding. This is the proof-of-concept code:

from functools import partial

if __name__ == '__main__':
sequence = ['foo', 'bar', 'spam']
loop_one = lambda seq: [lambda: el for el in seq]
no_op = lambda x: x
loop_two = lambda seq: [partial(no_op, el) for el in seq]
for func in (loop_one, loop_two):
print [f() for f in func(sequence)]


The output of the above is:

['spam', 'spam', 'spam']
['foo', 'bar', 'spam']


The behaviour of
loop_one
is surprising to me as I would expect it to behave as
loop_two
:
el
is an immutable value (a string) that changes at each loop, but
lambda
seems to store a pointer to the "looping variable"
, like if the loop would recycle the same memory address for each element of the sequence.

The above behaviour is the same with full-blown functions with a for loop in them (so it is not a list-comprehension syntax).

But wait: there is more... and more puzzling!

The following script works like
loop_one
:

b = []
for foo in ("foo", "bar"):
b.append(lambda: foo)

print [a() for a in b]


(output:
['bar', 'bar']
)

But watch what happens when one substitute the variable name
foo
with
a
:

b = []
for a in ("foo", "bar"):
b.append(lambda: a)

print [a() for a in b]


(output:
[<function <lambda> at 0x25cce60>, <function <lambda> at 0x25cced8>]
)

Any idea of what is happening here? I suspect there must be some gotcha related to the underlying C implementation of my interpreter, but I haven't anything else (Jthon, PyPy or similar) to test if this behaviour is consistent across different implementations.

Answer

The function lambda: el used in loop_one refers to a variable el which is not defined in the local scope. Therefore, Python looks for it next in the enclosing scope of the other lambda:

lambda seq: [lambda: el for el in seq]

in accordance with the so-called LEGB rule.

By the time lambda: el is called, this enclosing lambda has (of course) already been called and the list comprehension has been evaluated. The el used in the list comprehension is a local variable in this enclosing lambda. Its value is the one returned when Python looks for the value of el in lambda: el. That value for el is the same for all the different lambda: el functions in the list comprehension: it is the last value assigned to el in the for el in seq loop. Thus, el is always 'spam', the last value in seq.


You've already found one workaround, to use a closure such as your loop_two. Another way is to define el as a local variable with a default value:

loop_one = lambda seq: [lambda el=el: el for el in seq]
Comments