A.Fagrell A.Fagrell - 3 months ago 50
C++ Question

dlclose() not unloading .so-file which is linking to boost

If my app loads (using dlopen) a .so file, that is linking to the Boost Test Framework, I can't unload the so file. Without linking to boost it seems to be fine to unload it.

App file main.cpp:

#include <dlfcn.h>
#include <iostream>

int main()
{
auto sFileName = "./libtest_library.so";
auto handle = dlopen(sFileName, RTLD_LAZY | RTLD_LOCAL);

if (!handle)
std::cerr << "Error: " << dlerror() << std::endl;

auto closing = dlclose(handle);
while(1);
return 0;
}


Library .so file (libtest_library.so):

#include <iostream>
//#include "boost/test/unit_test.hpp"

static void con() __attribute__((constructor));
static void dcon() __attribute__((destructor));

void con()
{
std::cout << "Constructing library..." << std::endl;
}

void dcon()
{
std::cout << "Destructing library..." << std::endl;
}


Running this I get the output:

Constructing library...
Destructing library...


If I link to Boost's unit test framework in libtest_library.so I only get the
Constructing library...
output. dlclose(handle) returns 0 (which is success).

Currently linking against Boost v. 1.60.0, compiling with gcc 5_2_0 on Ubuntu 14.04.
Is this a bug in Boost? Compiler? Any ideas?

I need to reload the .so files several times in a project, and it needs to be fully unloaded (not existing in the memory). How can I solve this? Thanks.




Update 1:

It seems like if I only link to boost the libtest_library destructor is actually called but the boost_test_framework library is not unloaded. However, if I include "boost/test/unit_test.hpp", the destructor won't be called (libtest_library.so refuse to unload).

Update 2:

Looking through boost's sources I’ve found that there is a c++ singleton in boost causing the problem.

I can replicate the problem in a simplified version.
Basically if I add the following singleton to the libtest_library it doesn’t work (can’t unload the .so file):

alt 1

class Singleton
{
public:
static Singleton & getInstance() { static Singleton instance; return instance; }
private:
Singleton() {}
~Singleton() {}
};

static Singleton & singleton = Singleton::getInstance();


But using this works:

alt 2

class Singleton
{
public:
static Singleton & getInstance();
private:
Singleton() {}
~Singleton() {}
};

Singleton & Singleton::getInstance() { static Singleton instance; return instance; }

static Singleton & singleton = Singleton::getInstance();


I’ve tried the different GCC compilers and it leads to the same results. For me this seems to be bug?

Also the symbols are a bit different: doing
nm –C libtest_library.so | grep –i singleton
I get

alt 1 (not working one):

0000000000201460 u guard variable for Singleton::getInstance()::instance
0000000000201458 b singleton
0000000000000e66 W Singleton::getInstance()
0000000000000f08 W Singleton::Singleton()
0000000000000f08 W Singleton::Singleton()
0000000000000f1c W Singleton::~Singleton()
0000000000000f1c W Singleton::~Singleton()
0000000000201468 u Singleton::getInstance()::instance


And alt 2:

00000000002012f8 b guard variable for Singleton::getInstance()::instance
0000000000201300 b singleton
0000000000000bb0 T Singleton::getInstance()
0000000000000cec W Singleton::Singleton()
0000000000000cec W Singleton::Singleton()
0000000000000d00 W Singleton::~Singleton()
0000000000000d00 W Singleton::~Singleton()
0000000000201308 b Singleton::getInstance()::instance


Any Ideas?

Update 3

I've extracted out the part in boost that seems to create the problem and created a minimal example that demonstrate the problem:

main_app.cpp - main app

#include <dlfcn.h>
#include <iostream>

int main()
{
for(auto i = 0; i < 2; i++) {
auto sFileName = "./libtest_library.so";
auto handle = dlopen(sFileName, RTLD_LAZY | RTLD_LOCAL);

if (!handle) {
printf("Dlerror: %s\n", dlerror());
continue;
}

auto closing = dlclose(handle);
printf("Dlerror: %s\n", dlerror());
}

return 0;
}


main_lib.cpp - libtest_library.so

#include <iostream>

template<typename Derived>
class trivial_singleton_t {
public:
static Derived& instance() { static Derived the_inst; return the_inst; }
protected:
trivial_singleton_t() {}
~trivial_singleton_t() {}
};

class singleton_t : public trivial_singleton_t<singleton_t> {

private:
friend class trivial_singleton_t<singleton_t>;
singleton_t() {}
};

singleton_t & singleton = singleton_t::instance();

static void con() __attribute__((constructor));
static void dcon() __attribute__((destructor));

void con()
{
std::cout << "Constructing library..." << std::endl;
}

void dcon()
{
std::cout << "Destructing library..." << std::endl;
}


I get the following output:

Constructing library...
Dlerror: (null)
DLerror: (null)
Destructing library...


hence the library is only unloaded when main exists.

Answer

As noted in the question there are STB_GNU_UNIQUE symbols in the compiled binary files.

The problem is that a library that is using those symbols and loaded with dlopen will be flagged as NODELETE, and therefore persist between the dlopen/dlclose calls. See line 445 here: http://osxr.org:8080/glibc/source/elf/dl-lookup.c

https://sourceware.org/binutils/docs/binutils/nm.html, STB_GNU_UNIQUE or u:

The symbol is a unique global symbol. This is a GNU extension to the standard set of ELF symbol bindings. For such a symbol the dynamic linker will make sure that in the entire process there is just one symbol with this name and type in use.

These are created when variables/methods are created within a anonymous namespace or as static global variables.

The quickest solution is to force the compiler to not build those symbols as STB_GNU_UNIQUE using a linker flag --no-gnu-unique.

Unfortunately this didn't work for me since I didn't have a sufficiently recent linker, luckily I could rebuild gcc with the following configuration option: --disable-gnu-unique-object. Remember to also rebuild the boost library using the linker flag or new gcc.