DevNull DevNull - 2 months ago 6
C Question

C - Externs - Safe way to monitor value in LD_PRELOAD'ed library

Background



I help maintain a simple command-line tool,
diskmanager
, used to monitor poor disk performance, primarily due to too many operations/users concurrently using the same disk. My work involves maintaining a library,
libdisksupervisor.so
, that is occasionally used to "supervise" the disk manager program by launching it via:

LD_PRELOAD=/public/libdisksupervisor.so /sbin/diskmanager


The reason we do this is because the library and the application have very different release schedules, source can't be shared due to cross-NDAs in place, etc. To make our lives easier, the maintainers of
diskmanager
created a few
extern
variables in the application, and added some calls to "dummy" functions in a library (
libdonothing.so
) that they bundled with
diskmanager
.

When a call is made to
int dummy(void)
(normally found in
libdonothing.so
, but we intercept it via
LD_PRELOAD
'ing
libdisksupervisor.so
, which also includes the same function prototype), we know that
diskmanager
is in a state where we can safely read
extern int internalStatus
(located in
diskimager
) from within our own library. The code for
dummy()
is quite simple:

# In source for diskmanager
int internalStatus = (-1);

# In libdummy.so
int dummy(void) { return 0; }

# In libdisksupervisor.so
extern int internalStatus;
int dummy(void) { syslog(LOG_ERR, "State:%d", internalStatus);





Problem



So far, so good. A few months back, one of the maintainers for
diskmanager
did something silly, and removed
int internalStatus
from
diskmanager
causing our library to cause a segmentation fault when executing
LD_PRELOAD=/public/libdisksupervisor.so diskmanager
. A similar issue arose when a junior engineer was fumbling with GCC hidden attributes and changing some values to be
static
, and again resulting in a segfault.




Question



Is there any way, within our code in
libdisksupervisor.so
, we can test for the presence of these
extern
(from the perspective of our library) variables before proceeding, possibly via some cryptic linker or GCC magic? I know I could just throw
nm
or
objdump
at it as part of a pre-validation script, but we need to accomplish this within our library alone.

Thank you.

Answer

Is there any way, within our code in libdisksupervisor.so, we can test for the presence of these extern (from the perspective of our library) variables before proceeding, possibly via some cryptic linker or GCC magic?

You have a timing problem here. You in fact don't need to do anything special to test for the presence and visibility of those symbols at compile time, in the version of diskmanager that you link against. The issue arises when you attempt to use libdisksupervisor.so with a version of diskmanager that turns out at runtime to be incompatible.

I know I could just throw nm or objdump at it as part of a pre-validation script, but we need to accomplish this within our c library alone.

I am not aware of any approach that would work with the way you are running the program and would not be susceptible to being easily and accidentally foiled by diskmanager maintenance.

But perhaps there is a way involving changing how you run the program. If what you presently call libdisksupervisor.so provided a program entry point (i.e. main()) and you ran it directly, it could dlopen() diskmanager and check for the presence of the needed symbols via dlsym(). It could then transfer control to diskmanager's main() (also accessed via dlsym()). You can think of this as inserting a shim between the system's dynamic linker and diskmanager.


Update:

The good news is that I have a proof-of-concept demonstrating that it can be done (see below). The bad news is that enabling the main executable to be loaded as a shared library requires special build options, and it sounds like it could be troublesome to get the other side to build with such options. On the other hand, this approach allows them to control and document precisely which symbols are exposed to your side, and maybe that would serve as a suitable carrot.

Anyway, the POC consists of three C source files, two auxiliary files, and a Makefile:

dummy.c

int dummy(void) {
    return 0;
}

main.c

#include <stdio.h>

int dummy(void);

#ifndef BREAKME
int internalStatus = 42;
#endif

int main(int argc, char *argv[]) {
    printf("dummy() returns %d\n", dummy());
    return 0;
}

shim.c

#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>
#include <assert.h>

#define TARGET_PATH "./mainprog"
#define NOT_FOUND_STATUS 127
#define MISSING_SYM_STATUS 126

typedef int (*main_type)(int, char **);

static int *internalStatus_p;
#define internalStatus (*internalStatus_p);

int dummy(void) {
    return internalStatus;
}

#define LOAD_SYM(dso, name, var) do { \
    char *e_; \
    var = dlsym(dso, name); \
    e_ = dlerror(); \
    if (e_) { \
        fprintf(stderr, "%s\n", e_); \
        return MISSING_SYM_STATUS; \
    } \
} while (0)

int main(int argc, char *argv[]) {
    void *diskmanager_bin = dlopen(TARGET_PATH, RTLD_LAZY | RTLD_GLOBAL);
    char *error;
    main_type main_p;

    if (!diskmanager_bin) {
        fprintf(stderr, "Could not load " TARGET_PATH ": %s\naborting\n", dlerror());
        return NOT_FOUND_STATUS;
    } else {
        error = dlerror();
        assert(!error);
    }

    LOAD_SYM(diskmanager_bin, "internalStatus", internalStatus_p);
    LOAD_SYM(diskmanager_bin, "main", main_p);

    return main_p(argc, argv);
}

#undef LOAD_SYM

mainprog_dynamic

{
    main; internalStatus;
};

shim_dynamic

{
    dummy;
};

Makefile

# sources contributing to a shared library must be built with -fpic or -fPIC
CFLAGS = -fPIC -std=c99
LDFLAGS = 

SHLIB_LDFLAGS = -shared
SHLIB_EXTRALIBS = -lc

# Sources contributing to the main program should be built with -fpie or -fPIE
SHMAIN_CFLAGS = -fpie
# The main program must be linked with -pie
SHMAIN_LDFLAGS = -pie

DL_EXTRALIBS = -ldl

LIBDUMMY_SO_VER = 0
LIBDUMMY = libdummy.so.$(LIBDUMMY_SO_VER)

all: mainprog shim

mainprog: main.o $(LIBDUMMY) mainprog_dynamic
    $(CC) $(CFLAGS) $(SHMAIN_CFLAGS) $(LDFLAGS) $(SHMAIN_LDFLAGS) -Wl,--dynamic-list=mainprog_dynamic -o $@ $< $(LIBDUMMY) $(SHLIB_EXTRALIBS)

main.o: main.c
    $(CC) $(CPPFLAGS) $(CFLAGS) $(SHMAIN_CFLAGS) -c -o $@ $<

libdummy.so.$(LIBDUMMY_SO_VER): libdummy.so
    ln -sf $< $@

libdummy.so: dummy.o
    $(CC) -o $@ $(CFLAGS) $(LDFLAGS) $(SHLIB_LDFLAGS) -Wl,-soname,libdummy.so.$(LIBDUMMY_SO_VER) $^ $(SHLIB_EXTRALIBS)

shim: shim.o shim_dynamic
    $(CC) $(CFLAGS) $(LDFLAGS) -Wl,--dynamic-list=shim_dynamic -o $@ $< $(DL_EXTRALIBS)

test: all
    @echo "LD_LIBRARY_PATH=`pwd` ./mainprog :"
    @LD_LIBRARY_PATH=`pwd` ./mainprog
    @echo "LD_LIBRARY_PATH=`pwd` ./shim :"
    @LD_LIBRARY_PATH=`pwd` ./shim

clean:
    rm -f *.o *.so *.so.* mainprog shim

This models the situation you describe, where the function you want to override resides in a separate shared library. It assumes the GNU toolchain. Having successfully built the example (make all), you can make test for a demo:

$ make test
LD_LIBRARY_PATH=/tmp/dl ./mainprog :
dummy() returns 0
LD_LIBRARY_PATH=/tmp/dl ./shim :
dummy() returns 42

The *_dynamic files tell the linker about symbols in the two executables that should be included among the exported (dynamic) symbols, even though nothing in the link references them.

This approach does not allow the shim to refer to the main program's internalStatus variable directly, for then the shim would need to link the main program as a library, and it would be automatically loaded by the dynamic linker when the shim runs. References to variables are always bound immediately, so that would result in an error from the dynamic linker if internalStatus disappeared, outside the control of the shim.