joell joell - 9 months ago 52
C Question

How to handle hooked WSARecv

I'm working on a project that involves hooking

. I know how to hook this function, I mean its just the same as hooking another function. Anyway the hard part is when
is used to perform overlapped operations. The idea is that when an application receives data to intercept that and be possible to modify this, I'm using pipes for this. The native DLL tunnels all data to a managed 'server'. This processes the input etc and returns it back to the native DLL. This works great for
. However the hard part is when an application uses overlapped sockets.

So I need the received data first before I can process it, this is the hard part. How would I do something like this? I thought of this, but they both seem like a mess:

is called using the WSAOverlapped:
Create a new thread, use
and pass the
of the WSAOverlapped structure. When the event is signaled process the data to the managed server and pass the data to the program.

is called using the completion routine:
Create a new thread, modify the call to the original function with
to a new function. Use
to put the thread in an alertable state. When the OperationCompleted is called process the data and pass data back to the program.

I could post my code but I didn't write because it seems like a bad solution.. So there is not really a point for that.

I cannot think of a better solution and this seems horrible because when an application calls
a lot (for example a large server using overlapped sockets to handle lots of clients) it creates a new thread for every call and that just seems like a bad idea.

So how can I do such thing?

Answer Source

There's no need to create a thread for each overlapped IO call.

When overlapped operations are used, they either have an associated event (which you can safely ignore), a completion routine, or are associated with an I/O Completion port.

To handle the first two cases you should hook both WSARecv() and WSAGetOverlappedResult().

If you need to handle the last, you'll also need to hook GetQueuedCompletionStatus()

Now, when you get a call to WSARecv(), for the event case, you do nothing special there (except possibly save some information in relation to the lpOverlapped, eg. the buffer), and process the data in WSAGetOverlappedResult() (which the application must call to get the success/error and bytes transferred.)

If a completion routine is present, save the lpOverlapped and lpCompletionRoutine, and pass your own completion routine to the real WSARecv().

Your routine should process the data and call the original completion routine.

To handle the I/O completion port case, have WSARecv() save lpOverlapped and buffers etc., in GetQueuedCompletionStatus(), call the original, and if the returned overlapped structure matches, handle the data.

You should also note that overlapped operations may complete immediately, in which case the event isn't signaled, the completion routine isn't called, and (IIRC) no completion is queued on the IOCP.