I'm building a game where each player must program his bot. The key idea is that the player will program in C (or C++, or whatever compatible language), build a DLL and send this DLL to the server, so that no one can get his code. The problem is: how to make sure that he is not calling any illegal function? Like creating files or opening a socket. The DLL will be loaded with LoadLibrary and a function will be called. All interaction will happen with callback functions. A possible solution would be placing a empty kernel32.dll (and others) so that all winapi calls will fail. Is this safe and works on every case? Is there a better way to do it?
Please note that the player thread (the one how called the dll) must still be able to comunicate with the game, maybe with an open socket. On Linux this can be easily done with
Your best bet is to create a user with reduced privileges, which will allow you to control file access quite easily, and run the bot code in a sub-process running as that user.
If you also want to restrict network connections, it is also easy to setup a firewall so that the aforementionned process does not have the right to connect to external hosts.
If you need more control over which API calls you allow or not, there is a technique called 'API Interception via DLL Redirection' which is explained for example here:
(found via google)