hunse hunse - 1 month ago 5
Python Question

Why do list comprehensions write to the loop variable, but generators don't?

If I do something with list comprehensions, it writes to a local variable:

i = 0
test = any([i == 2 for i in xrange(10)])
print i


This prints "9". However, if I use a generator, it doesn't write to a local variable:

i = 0
test = any(i == 2 for i in xrange(10))
print i


This prints "0".

Is there any good reason for this difference? Is this a design decision, or just a random byproduct of the way that generators and list comprehensions are implemented? Personally, it would seem better to me if list comprehensions didn't write to local variables.

Answer

Python’s creator, Guido van Rossum, mentions this when he wrote about generator expressions that were uniformly built into Python 3: (emphasis mine)

We also made another change in Python 3, to improve equivalence between list comprehensions and generator expressions. In Python 2, the list comprehension "leaks" the loop control variable into the surrounding scope:

x = 'before'
a = [x for x in 1, 2, 3]
print x # this prints '3', not 'before'

This was an artifact of the original implementation of list comprehensions; it was one of Python's "dirty little secrets" for years. It started out as an intentional compromise to make list comprehensions blindingly fast, and while it was not a common pitfall for beginners, it definitely stung people occasionally. For generator expressions we could not do this. Generator expressions are implemented using generators, whose execution requires a separate execution frame. Thus, generator expressions (especially if they iterate over a short sequence) were less efficient than list comprehensions.

However, in Python 3, we decided to fix the "dirty little secret" of list comprehensions by using the same implementation strategy as for generator expressions. Thus, in Python 3, the above example (after modification to use print(x) :-) will print 'before', proving that the 'x' in the list comprehension temporarily shadows but does not override the 'x' in the surrounding scope.

So in Python 3 you won’t see this happen anymore.

Interestingly, dict comprehensions in Python 2 don’t do this either; this is mostly because dict comprehensions were backported from Python 3 and as such already had that fix in them.

There are some other questions that cover this topic too, but I’m sure you have already seen those when you searched for the topic, right? ;)

Comments