Python Question

Improving Python Threads Performance based on Resource Locking

The reason between Java and Python threads is

  • Java is designed to lock on the resources

  • Python is designed to lock the thread itself(GIL)

So Python's implementation performs better on a machine with singe core processor. This is fine 10-20 years before. With the increasing computing capacity, if we use multiprocessor machine with same piece of code, it performs very badly.

Is there any hack to disable GIL and use resource locking in Python(Like Java implementation)?

P.S. My application is currently running on Python 2.7.12. It is compute intensive with less I/O and network blocking. Assume that I can't use
for my use case.

Answer Source

I think the most straight way for you, that will give you also a nice performance increase is to use Cython.

Cython is a Python superset that compiles Python-like code to C code (which makes use of the cPython API), and from there to executable. It allows one to optionally type variables, that then can use native C types instead of Python objects - and also allows one direct control of the GIL.

It does support a with nogil: statement in which the with block runs with the GIL turned off - if there are other threads running (you use the normal Python threading library), they won't be blocked while code is running on the marked with block.

Just keep in mind that the GIL is there for a reason: it is thanks to it that global complex objects like lists and dictionaries work without the danger of getting into an inconsistent state between treads. But if your "nogil" blocks restrict themselves to local data structures, you should have no problems.

Check the Cython project - and here is an specific example of turning off the GIL: