Eric Sauer Eric Sauer - 2 months ago 9
Python Question

Multithreading performance overhead

So basically I created this program that adds values to redis. So far I get this timing:

real 0m27.759s
user 0m18.129s
sys 0m5.580s


However when I tried to run multiple threads:

if __name__ == '__main__':
try:
for x in range(0, NUM_THREADS):
Thread(None, startProgram, None,
(NUM_HOSTS/NUM_THREADS*x+1,
NUM_HOSTS/NUM_THREADS*(x+1))).start()
except Exception as errtxt:
print errtxt


I get this with
NUM_THREADS
set ot
10
:

real 0m32.642s
user 0m22.953s
sys 0m11.473s


Why is my program running slower with more threads?

I'm running Linux Ubuntu 11.04 and Python 2.7.1.

Answer

The result depends on the Python implementation, cpython's GIL prevents parallel computations from being faster than sequential ones.

Consider using the multiprocessing module, which executes each thread in its own Python process, or alternative GIL-free Python implementations like IronPython and Jython.