Bak Itzik Bak Itzik - 1 year ago 129
Python Question

Trying to get a grip on CachedPropery in clang\

This is related to other question I had, which left with no answer...
I trying to understand what's going on under the hood of the Python binding to libclang, and having really hard-time doing so.

I've read TONs of articles about both

in Python, in order to understand how the CachedProperty class in clang/ works, but still can't get all the pieces together.

The most related texts I've seen is one SO answer, and this code recipe in ActiveState. This helps me a bit, but - as I mentioned - I'm still not there.

So, let's cut to the chase:
I want to understand why am I getting
on creating CIndex. I will post here only the relevant code ( is 3646 lines long..), and I hope I don't miss anything that is relevant to me.
My code has only one relevant line, which is:

index = clang.cindex.Index.create()

This reffers to line 2291 in, which yields:

return Index(conf.lib.clang_createIndex(excludeDecls, 0))

From now on, there's a series of function calls, which I can't explain why and WTH did they come from. I'll list the code and
output along the questions that relevant to each part:

(Important thing to notice ahead: conf.lib defined like this:)

class Config:

def lib(self):
lib = self.get_cindex_library()
return lib

CachedProperty code:

class CachedProperty(object):
"""Decorator that lazy-loads the value of a property.

The first time the property is accessed, the original property function is
executed. The value it returns is set as the new value of that instance's
property, replacing the original method.

def __init__(self, wrapped):
self.wrapped = wrapped
self.__doc__ = wrapped.__doc__

def __get__(self, instance, instance_type=None):
if instance is None:
return self

value = self.wrapped(instance)
setattr(instance, self.wrapped.__name__, value)

return value


-> return Index(conf.lib.clang_createIndex(excludeDecls, 0))
(Pdb) s
> d:\project\clang\
-> def __get__(self, instance, instance_type=None):
(Pdb) p self
<clang.cindex.CachedProperty object at 0x00000000027982E8>
(Pdb) p self.wrapped
<function Config.lib at 0x0000000002793598>

  1. Why the next call after
    Index(conf.lib.clang_createIndex(excludeDecls, 0))
    is to
    method? What about the

  2. If the
    method isn't get called, how comes that self.wrapped has


(Pdb) r
> d:\project\clang\><CDLL 'libcla... at 0x27a1cc0>
-> return value
(Pdb) n
> c:\program files\python35\lib\ctypes\
-> def __getattr__(self, name):
(Pdb) r
> c:\program files\python35\lib\ctypes\><_FuncPtr obj...000000296B458>
-> return func

  1. Where
    should return value to? Where the call for
    method come from?


(Pdb) n
> d:\project\clang\
-> def __init__(self, obj):
(Pdb) p obj

This is the creation of
, which class Index inherits from.

  1. But - where there's any call to
    with one parameter? Is this is the one that
    conf.lib.clang_createIndex(excludeDecls, 0)

  2. Where is this number (40998256) coming from? I'm getting the same number over and over again. As far as I understand, it should be just a number, but a
    clang.cindex.LP_c_void_p object
    and that's why the assertion failed.

To sum it up, the best for me will be step-by-step guidance of the functions invocation over here, cause I'm felling a little lost in all this...

SOLUTION to the last 2 questions:
The problem lays in the difference between Python 2 and 3 in the
function. While Python 2 actually does the mapping, Python 3 only return iterator, which you can use later on. This results the
method on
to run without actually register any function - hence the wrong translation of the returning value.

Fix: Change map(register, functionList) to list(map(register, functionList))

Still, thanks @Martijn, because of you I was able to move on from this

Answer Source

The CachedProperty object is a descriptor object; the __get__ method is called automatically whenever Python tries to access an attribute on an instance that is only available on the class and has a __get__ method.

Using CachedProperty as a decorator means it is called and an instance of CachedProperty is created that replaces the original function object on the Config class. It is the @CachedProperty line that causes CachedProperty.__init__ to be called, and the instance ends up on the Config class as Config.lib. Remember, the syntax

def lib(self):
    # ...

is essentially executed as

def lib(self):
    # ...
lib = CachedProperty(lib)

so this creates an instance of CachedProperty() with lib passed in as the wrapped argument, and then Config.lib is set to that object.

You can see this in the debugger; one step up you could inspect type(config).lib:

(Pdb) type(config)
<class Config at 0x00000000027936E>
(Pdb) type(config).lib
<clang.cindex.CachedProperty object at 0x00000000027982E8>

In the rest of the codebase config is an instance of the Config class. At first, that instance has no lib name in the __dict__ object, so the instance has no such attribute:

(Pdb) 'lib' in config.__dict__

So trying to get config.lib has to fall back to the class, where Python finds the Config.lib attribute, and this is a descriptor object. Instead of using Config.lib directly, Python returns the result of calling Config.lib.__get__(config, Config) in that case.

The __get__ method then executes the original function (referenced by wrapped) and stores that in config.__dict__. So future access to config.lib will find that result, and the descriptor on the class is not going to be used after that.

The __getattr__ method is called to satisfy the next attribute in the conf.lib.clang_createIndex(excludeDecls, 0) expression; config.lib returns a dynamically loaded library from cdll.LoadLibrary() (via CachedProperty.__get__()), and that specific object type is handled by the Python ctypes libary. It translates attributes to specific C calls for you; here that's the clang_createIndex method; see Accessing functions from loaded dlls.

Once the call to conf.lib.clang_createIndex(excludeDecls, 0) completes, that resulting object is indeed passed to Index(); the Index() class itself has no __init__ method, but the base class ClangObject does.

Whatever that return value is, it has a representation that looks like an integer number. However, it almost certainly is not an int. You can see what type of object that is by using type(), see what attributes it has with dir(), etc. I'm pretty certain it is a ctypes.c_void_p data type representing a clang.cindex.LP_c_void_p value (it is a Python object that proxies for the real C value in memory); it'll represent as an integer:

Represents the C void * type. The value is represented as integer. The constructor accepts an optional integer initializer.

The rest of the clang Python code will just pass this value back to more C calls proxied by config.lib.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download