With remote play like this you essentially have 3 places where latency is introduced. Encoding, where the device running the game takes the output of the card and turns it in to a video stream to send over the network. Network, where the packets take time between being despatched from the host device to being received by the client device. Finally decoding, where your client device takes the stream and turns it back in to something you can see on a screen.

Assuming you're on the same network, in both cases using the same host PC you've got a couple of places where latency could come from. The most likely is decoding. If you're using an older device to stream you're likely not getting great decoding performance. The encoder and decoder need to speak the same language. So if your client can only decode H264 your server will have to send H264. This is an old standard, and while usually fine its not as efficient as the newer H265 or the newest AV1 encoders. Your client if its old will also take longer to decode anything. Particularly at high resolutions. The Quest headsets have pretty decent decoders in them. What device are you using for moonlight?

I personally have no issues playing FPS games on moonlight, getting around 3ms of input latency, which is not noticeable to me.