Dennis Dennis - 3 months ago 6x
Linux Question

Using r8 register as a loop counter results in endless loop - why?

The following code prints hello world 10 times by using the rsi register as a loop counter.

section .data
hello: db 'Hello world!',10
helloLen: equ $-hello

section .text
global _start

mov rsi, 0 ;<--- use r8 here

inc rsi ;<--- use r8 here

;print hello world
mov eax,4
mov ebx,1
mov ecx,hello
mov edx,helloLen

int 80h

cmp rsi, 10 ;<--- use r8 here
jnz do_loop

;system exit
mov eax,1 ; The system call for exit (sys_exit)
mov ebx,0 ; Exit with return code of 0 (no error)
int 80h;

If I am trying to use the r8 register instead of rsi as a loop counter it results in an endless loop. The r8 register here is just an example. It happens for the registers r9, r10 as well.

Could someone explain this, because I thought these are all general purpose registers and you should be allowed to use them?


TL;DR : int 0x80 implicitly zeroes out R8, R9, R10, and R11 on 64-bit Linux systems prior to returning to userland code. This behavior occurs on kernels later than 2.6.32-rc1. This is not the case for the preferred 64-bit SYSCALL calling convention.

You are experiencing a peculiarity of the Linux Kernels after version 2.6.32-rc1. For Linux kernel versions <= 2.6.32-rc1 you may get the behaviour you were expecting. Due to an information leakage bug (and exploits) the registers R8, R9, R10, and R11 are now zeroed out when the kernel returns from an int 0x80.

You may believe that these registers shouldn't matter in compatibility mode (32-bit code) since those newer registers are unavailable. This is a false assumption as it's possible for a 32-bit application to switch into 64-bit long mode and access those registers. The Linux Kernel Mailing List post that identified this issue had this to say:

x86: Don't leak 64-bit kernel register values to 32-bit processes

While 32-bit processes can't directly access R8...R15, they can gain access to these registers by temporarily switching themselves into 64-bit mode.

Code that demonstrates register leakage on the earlier kernels was made available by Jon Oberheide. It creates a 32-bit application to be run on an x86-64 system with IA32 compatibility enabled. The program switches to 64-bit long mode and then stores registers R8-R11 into general purpose registers that are available in compatibility mode (32-bit mode). John discusses the specifics in this article. He sums the vulnerability and the kernel fix nicely in this excerpt:

The Vulnerability

The underlying issue that causes this vulnerability is a lack of zeroing out several of the x86-64 registers upon returning from a syscall. A 32-bit application may be able to switch to 64-bit mode and access the r8, r9, r10, and r11 registers to leak their previous values. This issue was discovered by Jan Beulich and patched on October 1st. The fix is obviously to zero out these registers to avoid leaking any information to userspace.

If you were to step through your code in a debugger like GDB you should discover that R8 is in fact set to zero after int 0x80. Since it is your loop counter your program ends up in an endless cycle printing Hello world!.