lanoxx lanoxx - 1 month ago 10
Linux Question

Linux syscalls and errno

Context: I am trying to write a small C program with inline asm that should run under Linux on an x86_64 system and being compiled with gcc in order to better understand how syscalls work under Linux.

My question is: How are error numbers returned from a syscall (e.g. write) in this environment? I understand that when I use a library such as glibc, it takes care of saving the resulting error code in the global

errno
variable. But where is the error number stored when I call a syscall directly through inline assembler? Will it be stored inside a separate register, or will it be encoded in
%rax
?

Let take the write syscall on linux as an example:

When call
write
then after the syscall returns I find it stores
0xfffffffffffffff2
inside
%rax
, do I need to
somehow extract the error code from that?

If I have the error code number, where should I look to identify the actual error that occured? Lets say I get the number 5 returned, which header file do I need to consult to find the corresponding symbolic error name.

I am calling the write syscall like this:

asm ("mov $1,%%rax;"
"mov $1,%%rdi;"
"mov %1,%%rsi;"
"mov %2,%%rdx;"
"syscall;"
"mov %%rax,%0;"
: "=r" (result)
: "r" (msg), "r" (len)
: "%rdx", "%rsi", "%rax", "%rdi" /* EDIT: this is needed or else the registers will be overwritten */
);


with
result
,
msg
and
len
defined like so:

long result = 0;
char* msg = "Hello World\n";
long len = 12;

Answer

The linux syscall's convention is that they encode both the possible error code and the return value for successful call in the return value. It's just glibc or other C libraries's wrappers that they will set errno to the error code returned by the underline syscall, and the wrapper will return -1. Take the write as an example, the kernel does the error processing similar to this:

ssize_t write(int fd, ...) {
    if (fd is not valid)
         return -EBADF;
    return do_write(...);
}

So as you can see, the error code is just in the return value, and depending on the semantics, there is always a way to check if the syscall succeeded or not by comparing it to a value not possible for successful operation. For most syscalls, like the write one, that means check if it is negative.