How can I detach an executable file from its process using WinAPI (C/C++)?
I found an application that releases files from processes, Unlocker, but I don't know how that works.
You don't, normally. Windows pages in your executable on-demand. When you call a function which isn't in RAM yet, Windows loads it (and the whole page it resides on).
As a result, if you would detach the EXE and the process would call a function that's still on disk, the process would crash.
For applications on removable media, there's a special
/SWAPRUN:CD linker flag. It prevents the crash by making a copy of the whole executable.