Axel Borja Axel Borja - 23 days ago 10
Python Question

How to perform cython files compilation in parallel?

I would like to perform

cython
files compilation in parallel.

So, I take a look at
Cython.Build
source file, and find the following signature for
cythonize
function:

def cythonize(module_list, exclude=None, nthreads=0, aliases=None,
quiet=False, force=False, language=None,
exclude_failures=False, **options):


And the following comment about cythonize
nthreads
option:

"For parallel compilation, set the 'nthreads' option to the number of
concurrent builds."


So I tried to use this option in my
setup.py
file, like that:

from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils.extension import Extension

EXTENSIONS = [Extension(...)
...
Extension(...)]

setup(name='...',
...
ext_modules=cythonize(EXTENSIONS, nthreads=8),
...)


But my
.pyx
files are still compiled sequentially using 1 thread.

I do not understand what I am doing wrong here and how to use
nthreads
option to perform
cythonize
compilation in parallel ?

Answer

I finally found a solution to compile my cython file in parallel:

from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils.extension import Extension

NB_COMPILE_JOBS = 8

EXTENSIONS = [Extension(...)
              ...
              Extension(...)]

def setup_given_extensions(extensions):
    setup(name='...',
          ...
          ext_modules=cythonize(extensions),
          ...)

def setup_extensions_in_sequential():
    setup_given_extensions(EXTENSIONS)

def setup_extensions_in_parallel():
    cythonize(EXTENSIONS, nthreads=NB_COMPILE_JOBS)
    pool = multiprocessing.Pool(processes=NB_COMPILE_JOBS)
    pool.map(setup_given_extensions, EXTENSIONS)
    pool.close()
    pool.join()

if "build_ext" in sys.argv:
    setup_extensions_in_parallel()
else:
    setup_extensions_in_sequential()
Comments