I have a pure Python module and I want to rewrite some of submodules using Cython. Then I would like to add the new Cython submodules to the original Python module and make them available only as an option, meaning that cythoning the module is not compulsory (in which case the 'old' pure Python module should be used).
Here is an example:
python setup.py install
I am not sure about your
setup.py requirement (I don’t know why you would need that) but as for the runtime import issue, I wrote a decorator to do just that:
from __future__ import print_function from importlib import import_module from functools import wraps import inspect import sys MAKE_NOISE = False def external(f): """ Decorator that looks for an external version of the decorated function -- if one is found and imported, it replaces the decorated function in-place (and thus transparently, to would-be users of the code). """ f.__external__ = 0 # Mark func as non-native function_name = hasattr(f, 'func_name') and f.func_name or f.__name__ module_name = inspect.getmodule(f).__name__ # Always return the straight decoratee func, # whenever something goes awry. if not function_name or not module_name: MAKE_NOISE and print("Bad function or module name (respectively, %s and %s)" % ( function_name, module_name), file=sys.stderr) return f # This function is `pylire.process.external()`. # It is used to decorate functions in `pylire.process.*`, # each of which possibly has a native (Cython) accelerated # version waiting to be imported in `pylire.process.ext.*` # … for example: if in `pylire/process/my_module.py` you did this: # # @external # def my_function(*args, **kwargs): # """ The slow, pure-Python implementation """ # pass # # … and you had a Cython version of `my_function()` set up # in `pylire/process/ext/my_module.pyx` – you would get the fast # function version, automatically at runtime, without changing code. # # TL,DR: you'll want to change the `pylire.process.ext` string (below) # to match whatever your packages' structure looks like. module_file_name = module_name.split('.')[-1] module_name = "pylire.process.ext.%s" % module_file_name # Import the 'ext' version of process try: module = import_module(module_name) except ImportError: MAKE_NOISE and print("Error importing module (%s)" % ( module_name,), file=sys.stderr) return f MAKE_NOISE and print("Using ext module: %s" % ( module_name,), file=sys.stderr) # Get the external function with a name that # matches that of the decoratee. try: ext_function = getattr(module, function_name) except AttributeError: # no matching function in the ext module MAKE_NOISE and print("Ext function not found with name (%s)" % ( function_name,), file=sys.stderr) return f except TypeError: # function_name was probably shit MAKE_NOISE and print("Bad name given for ext_function lookup (%s)" % ( function_name,), file=sys.stderr) return f # Try to set telltale/convenience attributes # on the new external function -- this doesn't # always work, for more heavily encythoned # and cdef'd function examples. try: setattr(ext_function, '__external__', 1) setattr(ext_function, 'orig', f) except AttributeError: MAKE_NOISE and print("Bailing, failed setting ext_function attributes (%s)" % ( function_name,), file=sys.stderr) return ext_function return wraps(f)(ext_function)
… this lets you decorate functions as
@external – and they are replaced at runtime automatically with the Cython-optimized versions you’ve provided.
If you wanted to extend this idea to replacing entire Cythonized classes, it’d be straightforward to use the same logic in the
__new__ method of a metaclass (e.g. opportunistic find-and-replace in the optimized module).