pbible pbible - 4 months ago 16
C++ Question

How to properly wrap std::vector<std::size_t> with SWIG for Python? Problems with std::size_t

I'm trying to get

std::vector<std::size_t>
to work with SWIG. I need to provide a python interface to a c++ library.
std::vector
s of primitive types and objects are working fine but there is a problem with
std::size_t
.

I provide a MCVE on github here.

Main issue



Basically the problem is that
std::size_t
is not recognized and
std::vector<std::size_t>
is treated as
std::vector< int,std::allocator< int > > *
. When I try to specify the template, I get the following.

Using
%template(VecSize) std::vector<std::size_t>;
gives:

swig -c++ -python c_swig_vec_std_size.i
:0: Warning(490): Fragment 'SWIG_AsVal_std_size_t' not found.
:0: Warning(490): Fragment 'SWIG_From_std_size_t' not found.
g++ -fpic -c c_swig_vec_std_size_wrap.cxx -I/public/users/paul/dev/software/Python-2.7.11/Include -I/public/users/paul/dev/software/Python-2.7.11
c_swig_vec_std_size_wrap.cxx: In static member function ‘static int swig::traits_asval<long unsigned int>::asval(PyObject*, swig::traits_asval::value_type*)’:
c_swig_vec_std_size_wrap.cxx:4289: error: ‘SWIG_AsVal_std_size_t’ was not declared in this scope
c_swig_vec_std_size_wrap.cxx: In static member function ‘static PyObject* swig::traits_from<long unsigned int>::from(const swig::traits_from::value_type&)’:
c_swig_vec_std_size_wrap.cxx:4295: error: ‘SWIG_From_std_size_t’ was not declared in this scope
make: *** [c] Error 1


Minimum Example



Example c++ class



The following class is enough to show the functionality that I need. The
std::vector<int>
is included to show the intended behavior.

class_vec_std_size.hpp

#ifndef STD_SIZE_VEC
#define STD_SIZE_VEC

#include <vector>

class StdSizeVec{

public:
StdSizeVec(){
_myVec = std::vector<std::size_t>();
_myVec.push_back(1);
_myVec.push_back(2);

_myInts = std::vector<int>();
_myInts.push_back(1);
_myInts.push_back(2);
}

~StdSizeVec(){
_myVec.clear();
}

inline std::vector<std::size_t> getValues(){
return _myVec;
}

inline std::vector<int> getInts(){
return _myInts;
}

private:
std::vector<std::size_t> _myVec;
std::vector<int> _myInts;
};
#endif


Various attempts at the interface



a_swig_vec_std_size.i



%module a_swig_vec_std_size
%{
#include "class_vec_std_size.hpp"
%}
%include "class_vec_std_size.hpp"


Output

[paul@login-0-0 stack_swig]$ python
Python 2.7.11 (default, May 7 2016, 23:37:19)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from a_swig_vec_std_size import StdSizeVec
>>> ssv = StdSizeVec()
>>> vals = ssv.getValues()
>>> vals
<Swig Object of type 'std::vector< std::size_t > *' at 0x2ad7047be330>
>>> ints = ssv.getInts()
>>> ints
<Swig Object of type 'std::vector< int > *' at 0x2ad7047be780>
>>> exit()
swig/python detected a memory leak of type 'std::vector< int > *', no destructor found.
swig/python detected a memory leak of type 'std::vector< std::size_t > *', no destructor found.
[paul@login-0-0 stack_swig]$


This is the basic naive approach. The pointers are not useful in python and there are memory leak messages that we can not expose to the users of the interface.

b_swig_vec_std_size.i



%module b_swig_vec_std_size
%{
#include "class_vec_std_size.hpp"
%}
%include "std_vector.i"
%include "class_vec_std_size.hpp"


Output

[paul@login-0-0 stack_swig]$ python
Python 2.7.11 (default, May 7 2016, 23:37:19)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from b_swig_vec_std_size import StdSizeVec
>>> ssv = StdSizeVec()
>>> vals = ssv.getValues()
>>> vals
<Swig Object of type 'std::vector< std::size_t,std::allocator< std::size_t > > *' at 0x2aee17458330>
>>> ints = ssv.getInts()
>>> ints
<Swig Object of type 'std::vector< int,std::allocator< int > > *' at 0x2aee17458930>
>>> exit()
swig/python detected a memory leak of type 'std::vector< int,std::allocator< int > > *', no destructor found.
swig/python detected a memory leak of type 'std::vector< std::size_t,std::allocator< std::size_t > > *', no destructor found.


Using the correct "std_vector.i", SWIG knows more about the vector and allocators but still these pointers are not useful to client code in python and there are memory leak error messages.

c_swig_vec_std_size.i



This interface uses the correct
%template
directives like this answer. Here SWIG does not understand
std::size_t
as a template argument.

%module c_swig_vec_std_size
%{
#include "class_vec_std_size.hpp"
%}
%include "std_vector.i"
%template(VecInt) std::vector<int>;

// Does not compile
//%template(VecSize) std::vector<std::size_t>;
//
// Gives the following errors
//swig -c++ -python c_swig_vec_std_size.i
// :0: Warning(490): Fragment 'SWIG_AsVal_std_size_t' not found.
// :0: Warning(490): Fragment 'SWIG_From_std_size_t' not found.
// g++ -fpic -c c_swig_vec_std_size_wrap.cxx -I/public/users/paul/dev/software/Python-2.7.11/Include -I/public/users/paul/dev/software/Python-2.7.11
// c_swig_vec_std_size_wrap.cxx: In static member function ‘static int swig::traits_asval<long unsigned int>::asval(PyObject*, swig::traits_asval::value_type*)’:
// c_swig_vec_std_size_wrap.cxx:4289: error: ‘SWIG_AsVal_std_size_t’ was not declared in this scope
// c_swig_vec_std_size_wrap.cxx: In static member function ‘static PyObject* swig::traits_from<long unsigned int>::from(const swig::traits_from::value_type&)’:
// c_swig_vec_std_size_wrap.cxx:4295: error: ‘SWIG_From_std_size_t’ was not declared in this scope
// make: *** [c] Error 1

//The following compiles but does not work
%template(VecSize) std::vector<size_t>;

%include "class_vec_std_size.hpp"


Output

[paul@login-0-0 stack_swig]$ python
Python 2.7.11 (default, May 7 2016, 23:37:19)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from c_swig_vec_std_size import StdSizeVec
>>> ssv = StdSizeVec()
>>> vals = ssv.getValues()
>>> vals
<Swig Object of type 'std::vector< std::size_t,std::allocator< std::size_t > > *' at 0x2b286104bd80>
>>> ints = ssv.getInts()
>>> ints
(1, 2)
>>> exit()
swig/python detected a memory leak of type 'std::vector< std::size_t,std::allocator< std::size_t > > *', no destructor found.


Now the
std::vector<int>
is working properly but SWIG's
%template(VecSize) std::vector<size_t>;
(without
std::
) does not do the job.

Some internet digging



I found a few posts which offer some clues.

Feeling like this I found a 2006 post with the same problem

The std::vector::size_type wrapped as a pointer not an integer link had some helpful info but the problem is not exactly the same.

I found this primitives.i from the magnum.fe project, but thinking wishfully and importing primitives.i did not work for me.

After that I tried to implement the
SWIG_AsVal_std_size_t
and
SWIG_From_std_size_t
similar to their approach, but no luck.

hand rolled std_size_t.i



%fragment("SWIG_From_std_size_t", "header", fragment=SWIG_From_frag(std::size_t))
{
SWIGINTERNINLINE PyObject * SWIG_From_std_size_t(std::size_t value)
{
return PyInt_FromSize_t(value);
}
}


%fragment("SWIG_AsVal_std_size_t", "header")
{

SWIGINTERNINLINE bool SWIG_AsVal_std_size_t(PyObject* in, std::size_t& value)
{

// Get integer type
if(PyInt_Check(in)){
long unsigned int long_uint = PyLong_AsLong(in);
value = static_cast<std::size_t>(long_uint);
return true;
}else{
return false;
}

}
}

%fragment(SWIG_From_frag(std::size_t));
%fragment("SWIG_AsVal_std_size_t");


This was imported in d_swig_vec_std_size.i. but it does not compile.

%module d_swig_vec_std_size
%{
#include "class_vec_std_size.hpp"
%}
%include "std_vector.i"
%template(VecInt) std::vector<int>;

%include "std_size_t.i"
%template(VecSize) std::vector<std::size_t>;

%include "class_vec_std_size.hpp"


Here I get this.

swig -c++ -python d_swig_vec_std_size.i
g++ -fpic -c d_swig_vec_std_size_wrap.cxx -I/public/users/paul/dev/software/Python-2.7.11/Include -I/public/users/paul/dev/software/Python-2.7.11
d_swig_vec_std_size_wrap.cxx: In static member function ‘static int swig::traits_asval<long unsigned int>::asval(PyObject*, swig::traits_asval::value_type*)’:
d_swig_vec_std_size_wrap.cxx:4311: error: invalid initialization of reference of type ‘size_t&’ from expression of type ‘swig::traits_asval::value_type*’
d_swig_vec_std_size_wrap.cxx:4288: error: in passing argument 2 of ‘bool SWIG_AsVal_std_size_t(PyObject*, size_t&)’
make: *** [d] Error 1


makefile



PYTHON=/public/users/paul/dev/software/Python-2.7.11

all: a b c d

a:
swig -c++ -python a_swig_vec_std_size.i
g++ -fpic -c a_swig_vec_std_size_wrap.cxx -I${PYTHON}/Include -I${PYTHON}
g++ -g -fpic -shared a_swig_vec_std_size_wrap.o -o _a_swig_vec_std_size.so

b:
swig -c++ -python b_swig_vec_std_size.i
g++ -fpic -c b_swig_vec_std_size_wrap.cxx -I${PYTHON}/Include -I${PYTHON}
g++ -g -fpic -shared b_swig_vec_std_size_wrap.o -o _b_swig_vec_std_size.so

c:
swig -c++ -python c_swig_vec_std_size.i
g++ -fpic -c c_swig_vec_std_size_wrap.cxx -I${PYTHON}/Include -I${PYTHON}
g++ -g -fpic -shared c_swig_vec_std_size_wrap.o -o _c_swig_vec_std_size.so

d:
swig -c++ -python d_swig_vec_std_size.i
g++ -fpic -c d_swig_vec_std_size_wrap.cxx -I${PYTHON}/Include -I${PYTHON}
g++ -g -fpic -shared d_swig_vec_std_size_wrap.o -o _d_swig_vec_std_size.so


clean: clean_a clean_b clean_c clean_d

clean_a:
rm a_swig_vec_std_size_wrap.cxx a_swig_vec_std_size.py a_swig_vec_std_size_wrap.o _a_swig_vec_std_size.so

clean_b:
rm b_swig_vec_std_size_wrap.cxx b_swig_vec_std_size.py b_swig_vec_std_size_wrap.o _b_swig_vec_std_size.so

clean_c:
rm c_swig_vec_std_size_wrap.cxx c_swig_vec_std_size.py c_swig_vec_std_size_wrap.o _c_swig_vec_std_size.so

clean_d:
rm d_swig_vec_std_size_wrap.cxx d_swig_vec_std_size.py d_swig_vec_std_size_wrap.o _d_swig_vec_std_size.so


Program Versions



python version: Python 2.7.11
g++ version: g++ (GCC) 4.4.7
swig version: SWIG Version 1.3.40


Using a newer swig version (swig-3.0.10) gives the same result for me.

Summary



I suspect the answer may be along the lines of interface d somewhere, but I have had no luck so far. There could be a problem with how
std::size_t
is implemented differently depending on the architecture instead of having a fixed size. In any case, I would expect SWIG to be able to handle it. Am I missing something? I would like to find a solution that does not involve making changes to the C++ library (such as encapsulating std::size_t in a
struct
or using
int
instead).

Trying Jens Monk's Solution



namespace std {
%template(VecSize) vector<size_t>;
}


I get this:

[paul@login-0-0 stack_swig]$ make clean
rm a_swig_vec_std_size_wrap.cxx a_swig_vec_std_size.py a_swig_vec_std_size_wrap.o _a_swig_vec_std_size.so
rm b_swig_vec_std_size_wrap.cxx b_swig_vec_std_size.py b_swig_vec_std_size_wrap.o _b_swig_vec_std_size.so
rm c_swig_vec_std_size_wrap.cxx c_swig_vec_std_size.py c_swig_vec_std_size_wrap.o _c_swig_vec_std_size.so
rm d_swig_vec_std_size_wrap.cxx d_swig_vec_std_size.py d_swig_vec_std_size_wrap.o _d_swig_vec_std_size.so
[paul@login-0-0 stack_swig]$ make
swig -c++ -python a_swig_vec_std_size.i
g++ -fpic -c a_swig_vec_std_size_wrap.cxx -I/public/users/paul/dev/software/Python-2.7.11/Include -I/public/users/paul/dev/software/Python-2.7.11
g++ -g -fpic -shared a_swig_vec_std_size_wrap.o -o _a_swig_vec_std_size.so
swig -c++ -python b_swig_vec_std_size.i
g++ -fpic -c b_swig_vec_std_size_wrap.cxx -I/public/users/paul/dev/software/Python-2.7.11/Include -I/public/users/paul/dev/software/Python-2.7.11
g++ -g -fpic -shared b_swig_vec_std_size_wrap.o -o _b_swig_vec_std_size.so
swig -c++ -python c_swig_vec_std_size.i
g++ -fpic -c c_swig_vec_std_size_wrap.cxx -I/public/users/paul/dev/software/Python-2.7.11/Include -I/public/users/paul/dev/software/Python-2.7.11
g++ -g -fpic -shared c_swig_vec_std_size_wrap.o -o _c_swig_vec_std_size.so
swig -c++ -python -I/public/users/paul/dev/software/swig-3.0.10 d_swig_vec_std_size.i
g++ -fpic -c d_swig_vec_std_size_wrap.cxx -I/public/users/paul/dev/software/Python-2.7.11/Include -I/public/users/paul/dev/software/Python-2.7.11
g++ -g -fpic -shared d_swig_vec_std_size_wrap.o -o _d_swig_vec_std_size.so
[paul@login-0-0 stack_swig]$ python
Python 2.7.11 (default, May 7 2016, 23:37:19)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from d_swig_vec_std_size import StdSizeVec
>>> ssv = StdSizeVec()
>>> vals = ssv.getValues()
>>> vals
<Swig Object of type 'std::vector< std::size_t,std::allocator< std::size_t > > *' at 0x2aba7dd8bd80>
>>> ints - ssv.getInts()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'ints' is not defined
>>> ints = ssv.getInts()
>>> ints
(1, 2)
>>> exit()
swig/python detected a memory leak of type 'std::vector< std::size_t,std::allocator< std::size_t > > *', no destructor found.
[paul@login-0-0 stack_swig]$ cat d_swig_vec_std_size.i
%module d_swig_vec_std_size
%{
#include "class_vec_std_size.hpp"
%}
%include "std_vector.i"
%template(VecInt) std::vector<int>;

%include "std_size_t.i"

namespace std {
%template(VecSize) vector<size_t>;
}


%include "class_vec_std_size.hpp"

Answer

Try defining size_t for swig as shown here - http://www.swig.org/Doc1.3/SWIG.html#SWIG_nn20

%inline %{
typedef long unsigned int size_t;
%}

namespace std {
  %template(VecSize) vector<size_t>;
}