Capow Capow - 3 years ago 146
C Question

Using PyCapsule in Cython

Summary



I need to store a C struct inside a Python object for use in other parts boosted with Cython. I believe a PyCapsule is best suited for this purpose, but the results of my code is not what I am expecting. While the pointer address is correctly returned, the memory appears to have been freed.

Details



I am new to Cython and I am learning to use it to speed up part of my code. For the purpose of asking the question, I have simplified my code and an int is used instead of a struct.

I wrote CythonTest.pyx according to my understanding of the PyCapsule documentation and compiled it with setup.py using the standard command:

python setup.py build_ext --inplace


CythonTest.pyx

#cython: language_level=3

from cpython.pycapsule cimport PyCapsule_New, PyCapsule_IsValid, PyCapsule_GetPointer

class Test:
def __init__(self):
cdef int test = 10
cdef const char *name = "test"
self.vars = PyCapsule_New(<void *>&test, name, NULL)

# Print pointer address
print("{0:x}".format(<unsigned long long>test))

def peek(self):
cdef const char *name = "test"
if not PyCapsule_IsValid(self.vars, name):
raise ValueError("invalid pointer to parameters")
cdef int *test = <int *>PyCapsule_GetPointer(self.vars, name)
print(test[0])

# Print pointer address
print("{0:x}".format(<unsigned long long>test))


setup.py

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules=cythonize("CythonTest.pyx"))


Then, I run this with the following Python script.

from CythonTest import Test

test = Test()
print(test.vars)
test.peek()


The console prints out the following:

cbde7ebe70
<capsule object "test" at 0x0000027516467930>
0
cbde7ebe70


It appears that the pointer has been successfully stored in a PyCapsule and retrieved as indicated by the identical address. However, 0 is now stored inside the address instead of 10. I am aware that using an int may have caused it to be garbage collected and changed the nature of the problem, but the same issue is observed when using PyMem_Malloc as well.

So the question is: what is the correct way of using PyCapsule?

Environment




  • Compiler: Visual Studio Express 2015

  • Cython: 0.26

  • OS: Windows 10 (64 bit)

  • Python: 3.5.3

  • Spyder (IDE): 3.2.3


Answer Source

test is a local variable (in C) so does not exist beyond the end of the __init__ function, and therefore when you try to access it again in peek the memory has already been used for something else.

You can allocate memory for test of the heap instead, so that the variable persists as long as you need (you need to create a destructor to deallocate it though).

from libc.stdlib cimport malloc, free

# destructor
cdef void free_ptr(object cap):
   # This should probably have some error checking in
   # or at very least clear any errors raised once it's done
   free(PyCapsule_GetPointer(cap,PyCapsule_GetName(cap)))

class Test:
    def __init__(self):
       cdef int* test = malloc(sizeof(int)) 
       test[0] = 10
       cdef const char *name = "test"
       self.vars = PyCapsule_New(<void *>&test, name, &free_ptr)

       # etc
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download