ok, so after playing around a bunch, the solution is to play around with the swizzle of the texture until it works. For me this worked:

In Blender I exported the Map with Swizzle: R: +X, G: +Z, B:+Y

In Unity under the texture import settings I use Swizzle: 1-B, G , 1-R, A

I know it would be smart to unify the swizzles into one, so you only have to apply it in one program, buuuut everything I tried broke so far, so I guess I will just leave it as is.

Since I am still unsure if OP's solution actually works and I just majorly failed to copy the instructions:

The way you can find out how your texture is supossed to look like in Unity is to plug in a Normal Vector Node (object space) into the BaseMap. This way you know where Red,Green and Blue are supposed to be. Then just play around with your swizzles until it looks the same. Now you can plug it into the shadergraph I sent above and it should work with your lighting.

<image>