borrrden borrrden - 15 days ago 4
C++ Question

What is causing the value of this pointer to change from a simple function call?

A couple pictures are worth a couple thousand words in this case, so I will start with a description of what I am observing. Here is my first exhibit which I will refer to as "stack frame 1":

stack frame 1

Here I am passing in a parameter of type

FLValue
(which is a typedef to a pointer to a C++ type that is internal to the DLL that defines the function being called) to a function called
FLValue_AsString
. The value of the parameter at this point is 0x0486d10e.

Now note my second exhibit which I will call "stack frame 2":

stack frame 2

This stack frame is immediately above stack frame 1, and is the result of entering the function
FLValue_AsString
. The debugger stepped immediately in and did not take any inlined side routes as far as I can tell (this was run with optimizations all disabled). However, strangely, note that the value of
v
(the above
FLValue
) is now 0xedcf6fc2. This obviously results in a crash due to invalid memory access. What in the world just happened?

Some other information:


  • The compiler is set to compile functions as
    __cdecl
    in all projects in the solution

  • The function in question is defined inside an
    extern "C"
    block

  • LiteCore.dll (the DLL which exposes
    FLValue_AsString
    ) does not compile files directly, but rather is sort of a conglomerate which links together three static libraries and two other dlls, and defines a definition file of exposed symbols and so the function is actually defined inside of one of the static libraries inside.

  • This is not the only place that I observe this behavior, although the majority of calls function normally

  • If I manually change the memory address in the debugger to the one from stack frame 1, then I can continue without the invalid memory access

  • When compiled with Apple Clang on OS X, or GCC on Ubuntu, none of these problems are observed



EDIT Some information in text form from the screen shots:

Trace @ stack frame 1 (starting at main()):

C4Tests.exe!PerfTest::insertDocs(const _FLArray * docs) Line 141 C++
C4Tests.exe!`anonymous namespace'::____C_A_T_C_H____T_E_S_T____491::test() Line 501 C++
C4Tests.exe!Catch::NWayMethodTestCase<`anonymous namespace'::____C_A_T_C_H____T_E_S_T____491>::invoke() Line 29 C++
C4Tests.exe!Catch::TestCase::invoke() Line 7519 C++
C4Tests.exe!Catch::RunContext::invokeActiveTestCase() Line 6159 C++
C4Tests.exe!Catch::RunContext::runCurrentTest(std::basic_string<char,std::char_traits<char>,std::allocator<char> > & redirectedCout, std::basic_string<char,std::char_traits<char>,std::allocator<char> > & redirectedCerr) Line 6131 C++
C4Tests.exe!Catch::RunContext::runTest(const Catch::TestCase & testCase) Line 5951 C++
C4Tests.exe!Catch::runTests(const Catch::Ptr<Catch::Config> & config) Line 6297 C++
C4Tests.exe!Catch::Session::run() Line 6405 C++
C4Tests.exe!Catch::Session::run(int argc, const char * const * const argv) Line 6384 C++
C4Tests.exe!main(int argc, char * * argv) Line 10333 C++


Value of parameter being passed into
FLValue_AsString()
:


foo 0x0486d10e {_byte=0x0486d10e "FRem... } const fleece::Value *


Trace @ stack frame 2:

LiteCore.dll!FLValue_AsString(const fleece::Value * v) Line 56 C++
C4Tests.exe!PerfTest::insertDocs(const _FLArray * docs) Line 141 C++
C4Tests.exe!`anonymous namespace'::____C_A_T_C_H____T_E_S_T____491::test() Line 501 C++
C4Tests.exe!Catch::NWayMethodTestCase<`anonymous namespace'::____C_A_T_C_H____T_E_S_T____491>::invoke() Line 29 C++
C4Tests.exe!Catch::TestCase::invoke() Line 7519 C++
C4Tests.exe!Catch::RunContext::invokeActiveTestCase() Line 6159 C++
C4Tests.exe!Catch::RunContext::runCurrentTest(std::basic_string<char,std::char_traits<char>,std::allocator<char> > & redirectedCout, std::basic_string<char,std::char_traits<char>,std::allocator<char> > & redirectedCerr) Line 6131 C++
C4Tests.exe!Catch::RunContext::runTest(const Catch::TestCase & testCase) Line 5951 C++
C4Tests.exe!Catch::runTests(const Catch::Ptr<Catch::Config> & config) Line 6297 C++
C4Tests.exe!Catch::Session::run() Line 6405 C++
C4Tests.exe!Catch::Session::run(int argc, const char * const * const argv) Line 6384 C++
C4Tests.exe!main(int argc, char * * argv) Line 10333 C++
C4Tests.exe!invoke_main() Line 64 C++


Value of parameter received by
FLValue_AsString()
:


v 0xedcf6fc2 {_byte=0xedcf6fc2 } const fleece::Value *


EDIT 2: Another weird piece of information is that in 64-bit builds the debugger shows the memory as 0xcccccccccccccccc when actually the memory is fine and the program continues without problem.

EDIT 3: Compiler options:

C4Tests.exe:


/GS /TP /analyze- /W3 /Zc:wchar_t /Zi /Gm- /Od /Ob0 /Fd"C4Tests.dir\Debug\vc140.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS" /D "_DEBUG" /D "DEBUG" /D "C4DB_THREADSAFE" /D "SQLITE_OMIT_LOAD_EXTENSION" /D "C4_TESTS" /D "CMAKE_INTDIR=\"Debug\"" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope /RTC1 /GR /Gd /Oy- /MDd /Fa"Debug/" /EHsc /nologo /Fo"C4Tests.dir\Debug\" /Fp"C4Tests.dir\Debug\C4Tests.pch"


LiteCore.dll:


/GS /TP /analyze- /W3 /Zc:wchar_t /Zi /Gm- /Od /Ob0 /Fd"LiteCore.dir\Debug\vc140.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS" /D "_DEBUG" /D "DEBUG" /D "C4DB_THREADSAFE" /D "SQLITE_OMIT_LOAD_EXTENSION" /D "CMAKE_INTDIR=\"Debug\"" /D "LiteCore_EXPORTS" /D "_WINDLL" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope /RTC1 /GR /Gd /Oy- /MDd /Fa"Debug/" /EHsc /nologo /Fo"LiteCore.dir\Debug\" /Fp"LiteCore.dir\Debug\LiteCore.pch"


FleeceStatic:


/GS /TP /analyze- /W3 /Zc:wchar_t /Zi /Gm- /Od /Ob0 /Fd"FleeceStatic.dir\Debug\FleeceStatic.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS" /D "_DEBUG" /D "DEBUG" /D "CMAKE_INTDIR=\"Debug\"" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope /RTC1 /GR /Gd /Oy- /MDd /Fa"Debug/" /EHsc /nologo /Fo"FleeceStatic.dir\Debug\" /Fp"FleeceStatic.dir\Debug\FleeceStatic.pch"


Linking Options:

C4Tests.exe:


/OUT:"D:\Development\couchbase-lite-core\build_cmake\C\tests\Debug\C4Tests.exe" /MANIFEST /NXCOMPAT /PDB:"D:/Development/couchbase-lite-core/build_cmake/C/tests/Debug/C4Tests.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib" "....\Debug\LiteCore.lib" "....\vendor\SQLiteCpp\sqlite3\Debug\sqlite3.lib" "....\vendor\forestdb\Debug\forestdb.lib" "......\vendor\openssl\libs\windows\x86\libeay32.lib" /IMPLIB:"D:/Development/couchbase-lite-core/build_cmake/C/tests/Debug/C4Tests.lib" /DEBUG /MACHINE:X86 /SAFESEH /INCREMENTAL /PGD:"D:\Development\couchbase-lite-core\build_cmake\C\tests\Debug\C4Tests.pgd" /SUBSYSTEM:CONSOLE /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"C4Tests.dir\Debug\C4Tests.exe.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1


LiteCore.dll:


/OUT:"D:\Development\couchbase-lite-core\build_cmake\Debug\LiteCore.dll" /MANIFEST /NXCOMPAT /PDB:"D:/Development/couchbase-lite-core/build_cmake/Debug/LiteCore.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib" "Debug\LiteCoreStatic.lib" "vendor\fleece\Debug\FleeceStatic.lib" "vendor\sqlite3-unicodesn\Debug\SQLite3_UnicodeSN.lib" "vendor\SQLiteCpp\sqlite3\Debug\sqlite3.lib" "vendor\forestdb\Debug\forestdb.lib" "Ws2_32.lib" "..\vendor\openssl\libs\windows\x86\libeay32.lib" /IMPLIB:"D:/Development/couchbase-lite-core/build_cmake/Debug/LiteCore.lib" /DEBUG /DLL /MACHINE:X86 /SAFESEH /INCREMENTAL /PGD:"D:\Development\couchbase-lite-core\build_cmake\Debug\LiteCore.pgd" /SUBSYSTEM:CONSOLE /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"LiteCore.dir\Debug\LiteCore.dll.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1


FleeceStatic:


/OUT:"D:\Development\couchbase-lite-core\build_cmake\vendor\fleece\Debug\FleeceStatic.lib" /NOLOGO


EDIT 4: Let's dive down even further into the emitted assembly. Starting right before the call the
FLValue_ToString()
:

mov eax,dword ptr [ebp-1B4h] // move foo's address into eax
push eax
call _FLValue_AsString (0144AC9Eh)
jmp dword ptr [__imp__FLValue_AsString (014BD208h)]
jmp FLValue_AsString (0187DE00h)
push ebp
mov ebp,esp
sub esp,18h
mov eax,0CCCCCCCCh
mov dword ptr [ebp-18h],eax
mov dword ptr [ebp-14h],eax
mov dword ptr [ebp-10h],eax
mov dword ptr [ebp-0Ch],eax // this should be v (the passed value ?)
mov dword ptr [ebp-8],eax
mov dword ptr [ebp-4],eax
cmp dword ptr [v],0 // v == ebp+0Ch
je FLValue_AsString+34h (0187DE34h) // no jump here
lea eax,[ebp-0Ch] // eax is overwritten with a pointer to 0xcccccccc
push eax
mov ecx,dword ptr [v] // at this point v is garbage
call fleece::Value::asString (016E4DE5h)
jmp fleece::Value::asString (01882DB0h)
...
mov dword ptr [this],eax
mov dword ptr [this],ecx // Why the double move here? Both are invalid anyway...


So I guess this looks to be messed up calling conventions after all? It seems like a misuse of ecx instead of eax?

RESPONSE TO QUERY ABOUT FLSLICE:
Externally it is defined like this:


typedef struct {
const void *buf;
size_t size;
} FLSlice;


internally in the static library it is typedef'd to a struct with the same member variables, with the addition of some methods that are used for internal operations as seen here.

Answer

I'm no expert on Visual C++ calling conventions, but I strongly suspect that the difference in the declaration of FLSlice between the two modules has resulted in the caller expecting the return value to be passed by register and the callee expecting it to be passed in memory. That results in a disagreement over which of the arguments represents the pointer in question.

From X86 calling conventions on Wikipedia:

There are some variations in the interpretation of cdecl, particularly in how to return values. [...] Some compilers return simple data structures with a length of 2 registers or less in the register pair EAX:EDX, and larger structures and class objects requiring special treatment by the exception handler (e.g., a defined constructor, destructor, or assignment) are returned in memory. To pass "in memory", the caller allocates memory and passes a pointer to it as a hidden first parameter; the callee populates the memory and returns the pointer, popping the hidden pointer when returning.

Note in particular that the presence of a constructor can affect the calling convention.