GordonBGood GordonBGood - 1 year ago 93
Python Question

Is there a reason Python 3 enumerates slower than Python 2?

Python 3 appears to be slower in enumerations for a minimum loop than Python 2 by a significant margin, which appears to be getting worse with newer versions of Python 3.

I have Python 2.7.6, Python 3.3.3, and Python 3.4.0 installed on my 64-bit windows machine, (Intel i7-2700K - 3.5 GHz) with both 32-bit and 64-bit versions of each Python installed. While there is no significant difference in execution speed between 32-bit and 64-bit for a given version within its limitations as to memory access, there is a very significant difference between different version levels. I'll let the timing results speak for themselves as follows:

C:\**Python34_64**\python -mtimeit -n 5 -r 2 -s"cnt = 0" "for i in range(10000000): cnt += 1"
5 loops, best of 2: **900 msec** per loop

C:\**Python33_64**\python -mtimeit -n 5 -r 2 -s"cnt = 0" "for i in range(10000000): cnt += 1"
5 loops, best of 2: **820 msec** per loop

C:\**Python27_64**\python -mtimeit -n 5 -r 2 -s"cnt = 0" "for i in range(10000000): cnt += 1"
5 loops, best of 2: **480 msec** per loop

Since the Python 3 "range" is not the same as Python 2's "range", and is functionally the same as Python 2's "xrange", I also timed that as follows:

C:\**Python27_64**\python -mtimeit -n 5 -r 2 -s"cnt = 0" "for i in **xrange**(10000000): cnt += 1"
5 loops, best of 2: **320 msec** per loop

One can easily see that version 3.3 is almost twice as slow as version 2.7 and Python 3.4 is about 10% slower than that again.

My question: Is there an environment option or setting that corrects this, or is it just inefficient code or the interpreter doing more for the Python 3 version?

The answer seems to be that Python 3 uses the "infinite precision" integers that used to be called "long" in Python 2.x its default "int" type without any option to use the Python 2 fixed bit-length "int" and it is processing of these variable length "int"'s that is taking the extra time as discussed in the answers and comments below.

It may be that Python 3.4 is somewhat slower than Python 3.3 because of changes to memory allocation to support synchronization that slightly slow memory allocation/deallocation, which is likely the main reason that the current version of "long" processing runs slower.

Answer Source

The difference is due to the replacement of the int type with the long type. Obviously operations with long integers are going to be slower because the long operations are more complex.

If you force python2 to use longs by setting cnt to 0L the difference goes away:

$python2 -mtimeit -n5 -r2 -s"cnt=0L" "for i in range(10000000): cnt += 1L"
5 loops, best of 2: 1.1 sec per loop
$python3 -mtimeit -n5 -r2 -s"cnt=0" "for i in range(10000000): cnt += 1"
5 loops, best of 2: 686 msec per loop
$python2 -mtimeit -n5 -r2 -s"cnt=0L" "for i in xrange(10000000): cnt += 1L"
5 loops, best of 2: 714 msec per loop

As you can see on my machine python3.4 is faster than both python2 using range and using xrange when using longs. The last benchmark with python's 2 xrange shows that the difference in this case is minimal.

I don't have python3.3 installed, so I cannot make a comparison between 3.3 and 3.4, but as far as I know nothing significant changed between these two versions (regarding range), so the timings should be about the same. If you see a significant difference try to inspect the generated bytecode using the dis module. There was a change about memory allocators (PEP 445) but I have no idea whether the default memory allocators were modified and which consequences there were performance-wise.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download