bereal bereal - 1 year ago 151
Python Question

Pass a closure from Cython to C++

I have a C++ function that accepts a callback, like this:

void func(std::function<void(A, B)> callback) { ... }

I want to call this function from Cython by giving it a closure, i.e. something I would have done with a lambda if I was calling it from C++. If this was a C function, it would have some extra

typedef void(*callback_t)(int, int, void*);

void func(callback_t callback, void *user_data) {
callback(1, 2, user_data);

and then I would just pass
(there is a more detailed example here).

Is there way to do this more in C++ way, without having to resort to explicit

Answer Source

What I believe you're aiming to do is pass a callable Python object to something accepting a std::function. You need to do create a bit of C++ code to make it happen, but it's reasonably straightforward.

Starting by defining "accepts_std_function.hpp" as simply as possible to provide an illustrative example:

#include <functional>
#include <string>

inline void call_some_std_func(std::function<void(int,const std::string&)> callback) {

The trick is then to create a wrapper class that holds a PyObject* and defines operator(). Defining operator() allows it to be converted to a std::function. Most of the class is just refcounting. "py_obj_wrapper.hpp":

#include <Python.h>
#include <string>
#include "call_obj.h" // cython helper file

class PyObjWrapper {
    // constructors and destructors mostly do reference counting
    PyObjWrapper(PyObject* o): held(o) {

    PyObjWrapper(const PyObjWrapper& rhs): PyObjWrapper(rhs.held) { // C++11 onwards only

    PyObjWrapper(PyObjWrapper&& rhs): held(rhs.held) {
        rhs.held = 0;

    // need no-arg constructor to stack allocate in Cython
    PyObjWrapper(): PyObjWrapper(nullptr) {

    ~PyObjWrapper() {

    PyObjWrapper& operator=(const PyObjWrapper& rhs) {
        PyObjWrapper tmp = rhs;
        return (*this = std::move(tmp));

    PyObjWrapper& operator=(PyObjWrapper&& rhs) {
        held = rhs.held;
        rhs.held = 0;
        return *this;

    void operator()(int a, const std::string& b) {
        if (held) { // nullptr check 
            call_obj(held,a,b); // note, no way of checking for errors until you return to Python

    PyObject* held;

This file uses a very short Cython file to do the conversions from C++ types to Python types. "call_obj.pyx":

from libcpp.string cimport string

cdef public void call_obj(obj, int a, const string& b):

You then just need to create the Cython code wraps these types. Compile this module and call test_func to run this. ("simple_version.pyx":)

cdef extern from "py_obj_wrapper.hpp":
    cdef cppclass PyObjWrapper:
        PyObjWrapper(object) # define a constructor that takes a Python object
             # note - doesn't match c++ signature - that's fine!

cdef extern from "accepts_std_func.hpp":
    void call_some_std_func(PyObjWrapper) except +
            # here I lie about the signature
            # because C++ does an automatic conversion to function pointer
            # for classes that define operator(), but Cython doesn't know that

def example(a,b):

def test_call():
    cdef PyObjWrapper f = PyObjWrapper(example)


The above version works but is somewhat limited in that if you want to do this with a different std::function specialization you need to rewrite some of it (and the conversion from C++ to Python types doesn't naturally lend itself to a template implementation). One easy way round this is to use the Boost Python library object class, which has a templated operator(). This comes at the cost of introducing an extra library dependency.

First defining the header "boost_wrapper.hpp" to simplify the conversion from PyObject* to boost::python::object

#include <boost/python/object.hpp>

inline boost::python::object get_as_bpo(PyObject* o) {
    return boost::python::object(boost::python::handle<>(boost::python::borrowed(o)));

You then just need to Cython code to wrap this class ("boost_version.pyx"). Again, call test_func

cdef extern from "boost_wrapper.hpp":
    cdef cppclass bpo "boost::python::object":
        # manually set name (it'll conflict with "object" otherwise

    bpo get_as_bpo(object)

cdef extern from "accepts_std_func.hpp":
    void call_some_std_func(bpo) except + # again, lie about signature

def example(a,b):

def test_call():
    cdef bpo f = get_as_bpo(example)


A ""

from distutils.core import setup, Extension
from Cython.Build import cythonize

extensions = [
           "simple_version",                       # the extension name
           sources=["simple_version.pyx", "call_obj.pyx" ],
           language="c++",                        # generate and compile C++ code
           "boost_version",                       # the extension name
           language="c++",                        # generate and compile C++ code

setup(ext_modules = cythonize(extensions))

(A final option is to use ctypes to generate a C function pointer from a Python callable. See Using function pointers to methods of classes without the gil (bottom half of answer) and I'm not going to go into detail about this here.)

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download