ravelinx ravelinx - 3 months ago 21
C Question

Get returned char from a C function using inline assembly

I have this function that uses inline assembly that basically calls a C function, gets the returned value, and passes that value as a parameter to another function that returns a character.

void convertText(FILE *arch, FILE *result)
int i = 0;
int n = arch->size;

_asm {
mov esi, 0
cmp esi, n
jge end

mov ebx, result
mov ebx, [ebx]result.information ; Pointer to an array of characters

push esi ; Push parameters to get5bitsFunc
push arch ; Push parameters to get5bitsFunc

call get5bitsFunc
pop arch ; Restore values
pop esi ; Restore values

push eax ; push get5bitsFunc returned value to codify as parameter
call codify
mov edi, eax ; <- HERE move returned value from codify to edi register
pop eax ; restore eax

inc esi
jmp whileS


Think of codify as function of the type

unsigned char codify(unsigned char parameter) {
unsigned char resp;

// Do something to the parameter
resp = 'b'; // asign value to resp
return resp;

I have already tested codify and works fine returning the value I want using C code. The problem is that when I run and debug the convertText code in inline assembly in the line I have marked as "-> Here" the value returned in eax is something of the type 3424242 and not 97 or above in the ascii table that is what I need.

How can I get the char value?


The Windows ABI apparently doesn't require functions returning char to zero- or sign-extend the value into EAX, so you need to assume that the bytes above AL hold garbage. (This is the same as in the x86 and x86-64 System V ABI. See also the tag wiki for ABI/calling convention docs).

You can't assume that zeroing EAX before calling codify() is sufficient. It's free to use all of EAX as a scratch register before returning with the char in AL, but garbage in the rest of EAX.

You actually need to movzx esi, al, (or MOVSX), or mov [mem], al or whatever else you want to do to ignore garbage in the high bytes.