lanoxx lanoxx - 9 months ago 68
Linux Question

Linux syscalls and errno

Context: I am trying to write a small C program with inline asm that should run under Linux on an x86_64 system and being compiled with gcc in order to better understand how syscalls work under Linux.

My question is: How are error numbers returned from a syscall (e.g. write) in this environment? I understand that when I use a library such as glibc, it takes care of saving the resulting error code in the global

errno
variable. But where is the error number stored when I call a syscall directly through inline assembler? Will it be stored inside a separate register, or will it be encoded in
%rax
?

Let take the write syscall on linux as an example:

When call
write
then after the syscall returns I find it stores
0xfffffffffffffff2
inside
%rax
, do I need to
somehow extract the error code from that?

If I have the error code number, where should I look to identify the actual error that occured? Lets say I get the number 5 returned, which header file do I need to consult to find the corresponding symbolic error name.

I am calling the write syscall like this:

asm ("mov $1,%%rax;"
"mov $1,%%rdi;"
"mov %1,%%rsi;"
"mov %2,%%rdx;"
"syscall;"
"mov %%rax,%0;"
: "=r" (result)
: "r" (msg), "r" (len)
: "%rdx", "%rsi", "%rax", "%rdi" /* EDIT: this is needed or else the registers will be overwritten */
);


with
result
,
msg
and
len
defined like so:

long result = 0;
char* msg = "Hello World\n";
long len = 12;

Answer Source

The linux syscall's convention is that they encode both the possible error code and the return value for successful call in the return value. It's just glibc or other C libraries's wrappers that they will set errno to the error code returned by the underline syscall, and the wrapper will return -1. Take the write as an example, the kernel does the error processing similar to this:

ssize_t write(int fd, ...) {
    if (fd is not valid)
         return -EBADF;
    return do_write(...);
}

So as you can see, the error code is just in the return value, and depending on the semantics, there is always a way to check if the syscall succeeded or not by comparing it to a value not possible for successful operation. For most syscalls, like the write one, that means check if it is negative.