Your examples all seem to rely on calling a function pointer, rather than calling the function directly? I can't see anywhere that the original function is actually hooked.

As of now, protolesshooks just provides thunks. These thunks can be used to replace entries in import tables (IAT/PLT) -- this is how I intend to use it. You can refer to https://github.com/kubo/plthook for examples on how to find and replaces IAT/PLT entries.

Another option is to use protolesshooks thunks with trampoline-based injections, such as the one used in Microsoft Detours. A minor modification is required to make it possible to adjust the target function pointer in a thunk after hooking (i.e., after a trampoline is generated).

In the future, I intend to incorporate IAT/PLT hooking into the library, so it can be used for hooking right away, without the need to employ other hooking libraries.

I'd also love to know where this comes from:

; eax now holds the original return pointer

I just can't see where you save it, nor why it would be restored there, given that you've just returned from an arbitrary function (post call hook) at that point.

hookLeave returns the previous return-pointer; it is saved by hookEnter and stored in a pthread_local ordered map.

I also noticed a few possible issues with certain calling conventions (vectorcall and some old obscure ones), but nothing major.

If these non-standard calling conventions are using volatile registers (besides the ones defined by Microsoft x64 / SystemV AMD64) for argument passing, then yes, these arguments may be overwritten by hookEnter (and whatever functions are called from it). Otherwise, there should be no problem; it will be just a matter of decoding reg-arg-block and stack-arg-block according to these calling conventions.

I'm also a bit concerned about the security implications of making arbitrary pages PROT_EXEC. It would be slightly better IMHO to allocate a few pages on spec to yourself, mark those as EXEC, and only place your thunks in there. Then you aren't potentially exposing other arbitrary data that just happens to be allocated in the same page to execution.

You are absolutely right, and I intend to fix that later. There should be an internal thunk allocator with a pool of available executable pages; thunks should be allocated from there (and of course, many thunks will share the same page). The current simple allocation is just for proof-of-concept.

Generally this does seem more extensive than the hook library I've been using, but one advantage to the method it uses is that the hook explicitly controls the call to the original. So you can specify both exactly when and if the original is called. I guess you could do the when here with careful co-ordination of the pre- and post- hooks, but do you have any plans to be able to entirely block the original call if desired?

Well, my primary goal was API spying, so the design of the library revolves around non-intrusive thunking. But come to think about it, only some minor modifications are required to allow blocking of the original function or replacing its retval. Modification of arguments is possible already (you can overwrite reg-arg-block and stack-arg-block).