sashoalm sashoalm - 3 months ago 13
Python Question

Py_InitModule copies the name but not the function pointer?

When I register callbacks using

Py_InitModule
, and if I later change the function pointer in the structure to point to a new function, the new function is called. But if I change the name, the new name is not recognized.

#include <Python.h>

PyObject* foo1(PyObject *self, PyObject *args)
{
printf("foo1\n");
Py_RETURN_NONE;
}

PyObject* foo2(PyObject *self, PyObject *args)
{
printf("foo2\n");
Py_RETURN_NONE;
}

int main()
{
PyMethodDef methods[] = {
{ "foo", foo1, METH_VARARGS, "foo" },
{ 0, 0, 0, 0 }
};

Py_Initialize();
Py_InitModule("foo", methods);
PyRun_SimpleString("import foo\n");
PyRun_SimpleString("foo.foo()\n");
methods[0].ml_meth = foo2;
PyRun_SimpleString("foo.foo()\n");
methods[0].ml_name = "foo2";
PyRun_SimpleString("foo.foo()\n");
PyRun_SimpleString("foo.foo2()\n");
return 0;
}


This gives the following output:

foo1
foo2
foo2
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'module' object has no attribute 'foo2'


This seems a very inconsistent behavior. I first encountered it when I used a stack variable for
PyMethodDef methods
, which crashed the program once the variable went out of scope and I still tried to call the C++ callback from python. So I tested that changing the pointer indeed changes which function is called even though I haven't re-registered it with another
Py_InitModule
call. But at the same time, changing the name does not have this behavior.

So far I'm pretty certain that
PyMethodDef
needs to live for as long as python code tries to call the methods (i.e. can't be stack/local variable), but only the function pointers themselves are used.

Is this an intentional behavior, or some oversight? The documentation doesn't mention anything about PyMethodDef lifetime that I could find.

Answer

The inconsistency you see arises from the difference between function code, which is the property of a function, and the function name, which is merely the key in the module's dict, and not the fundamental property of the function. This is quite intentional, and allows using the same function in many places, which would not be possible if you could "rename" it at will.

One can demonstrate the same difference using regular Python functions, like this:

>>> def add(a, b): return a + b
... 
>>> def sub(a, b): return a - b
... 
>>> add
<function add at 0x7f9383127938>  # function looks like it has a name
>>> add.__name__ = 'foo'
>>> add                           # the name looks changed, but...
<function foo at 0x7f9383127938>
>>> foo                           # the change doesn't affect the module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'foo' is not defined
>>> add.__code__ = sub.__code__   # ...but we can still change the code
>>> add(2, 2)
0

As for your question in the comment: the method fields are not copied because Py_InitModule and the related functions are designed to be called with statically allocated variables, and creating many copies would be a waste of space for the common case. Not copying them explains why changing the actual C callback in ml_meth changes the Python callable.