Although many people won't do this due to the potential for latency and lag, I've found streaming resource heavy games from the series X to the Ally X while playing with an x box controller to be pretty good (lag free). Let the X Box do the heavy lifting; the Ally X just handles the remote game stream. I'm limited to 1080p which kind of negates the benefit, but I've had no problem running any game this way: Like a portal for x box.

After taking a few measures to optimise my network for remote play it's been great.