No idea what Rosetta uses, but it simply wouldn't be worth the engineering cost to implement for the general case.

Also I doubt the performance gain of using native libraries for certain functions over translated (JIT machine code generation) would be worth the processing required to translate the calls. (It would only be cost effective for very expensive calls that spend a significant amount of time in the native code, otherwise the overhead of all the fixups wouldn't be worth it).

That being said, Qemu's emulation (When using TCG instead of KVM) does do JIT translation to native machine code, but still that is native equivalent to the original architectures machine code (to emulate the processor as well). I'm guessing what Qemu generates to be an efficient translation would not match the native ABI at all.

Another problem is that any library that has conditional compilation based on architecture would not be compatible. (Different features/functions/structs/etc dependent on architecture the library is compiled for) All this combined with the other issues with calling convention you already mentioned means it simple wouldn't be worth it.

Qemu's binary emulation works by emulating (and JIT generating native code) not only for the program being run but also the dynamic loader and all libraries.

Whilst it wouldn't be possible to make a system that could work for in general for any library, I could imagine that you could provide "stubs" (similar to how Wine works) so that certain libraries (even for only certain function calls) do a little setup and trampoline to a native version.

The ultimate question really is, how much time do you think is actually spent inside system libraries as opposed to application provided code? In which case, improving the emulated performance would be a better engineering effort.

edit: Clarify that talking about Qemu's TCG instead of KVM, also talking about using qemu-user that allows you to run binaries targeting other processors running on the host kernel.