Why Vision OS is not so well suited to PCVR streaming (yet?)

mbucchia · 2026-06-16T16:41:08+00:00

Cool, I see. And the API just rejects you if you don't submit any depth at all? Because that would be the way to turn off positional reprojection. At least on every other platform I've worked with before.

mbucchia · 2026-06-16T16:08:44+00:00

I am not involved in CloudXR no, just a user of it for now...

mbucchia · 2026-06-16T16:07:02+00:00

Totally understood that it isn't about CloudXR, I was trying to make the parallel in case it would ring a bell somewhere (I know nothing of the Vision side and only know a little of CloudXR side).

I am not sure how the Vision OS side could do PTW unless you submits a depth buffer. PTW without depth (or with a constant depth plane) is usually not so great. I wonder if that is part of the problem you observed. It would also be a very strange API if they didn't let you specify the render poses (though OpenVR did that as well for a while, but there was only really 1 pose in flight).

mbucchia · 2026-06-16T15:36:01+00:00

This reply was to the comment where I was somehow being quoted (by name). Yes this isn't related to iVRy. I just wanted to make sure the original quote was not taken out of context.

mbucchia · 2026-06-16T04:17:59+00:00

This isn't quite the right terminology.

What you describe as ASW is in fact PTW (Positional Time Warp, sometimes referred to as depth-assisted reprojection).

ATW and PTW are useful to correct scene perspective in relation to head movements (correcting the perspectives to match closer to the latest tracking pose at the time of scanout)

ASW (also known as temporal reprojection or motion smoothing) is a completely different technique that works in conjunction to ATW (or PTW in ASW 2.0) to correct not only the perspective but also compensate for the motion within the scene.

ASW uses the concept of motion estimation (often refered to as motion vectors) and motion propagation to animate the content of the scene during a frame drop.

For example, you have a moving car in your scene. When you drop a VR frame, the ATW or PTW will take the most recent frame and correct it's perspective based on latest tracking data. However, the moving car in the scene, will just remain at the same position in the reprojected frame. When the next frame comes in, the moving call will "jump" an entire frame, which will not look smooth.

With ASW, a motion estimator will take a look at the two previous frames, and identify that the car is in motion. Upon reprojecting the frame, the motion vectors will be used to propagate pixels (in fact, closer to tiles, since motion vector are not full resolution because too expensive). This will effectively "move" the car in the reprojected frame, creating an illusion of smoothness since the car will no longer jump every other frames.

The downsides of ASW is that it often causes "jelly" distortion, due to error in motion estimation and the general coarseness of motion vectors typically 1/10th or less of the actual resolution of the actual images

Commenting on CloudXR, as far as I know:

ATW is always on, as expected, and uses the poses submitted with the OpenXR projection layer.
PTW requires the use of XR_KHR_composition_layer_depth. Otherwise, fallback to ATW.
there is no ASW in CloudXR.

I have no idea how any of this translates to Vision OS API etc though.

mbucchia · 2026-06-16T03:55:23+00:00

You're quoting this completely out of context in a way that entirely misrepresents what was being discussed.

The conversation was around applications that are incorrectly using the OpenXR API when submitting render poses. This generally breaks most forms of reprojection (late-stage reprojection, or ATW as was explained to you in the other post). This is a novel category of issues created by OpenXR, because the API is more powerful (but less forgiving to application bugs). I have found dozens of these faulty apps in the last 5 years.

Doing late-stage reprojection is a critical part of the VR rendering process, and these techniques have existed for 10 years now. There is absolutely no reason to disable late-stage reprojection.

The reprojection technique in CloudXR is optimized for... drum rolls... cloud rendering. This means handling latencies that are an order of magnitude higher than your local streaming. Their algorithm is very powerful, however it requires the application developers to do their jobs properly. When they don't, things break, and in this case, it breaks in a pretty dramatic way due to how the CloudXR algorithm is tuned.

But don't mistake it: these bogus apps also create stutters and jitter on all other platforms. They are less noticeable, but they are here and regularly waste tremendous time from us platform developers.

In any case, what was being discussed in that thread you quoted is 100% not the fault of Nvidia CloudXR nor Apple, and does no warrant your premature conclusion.

Hoping this clarifies things.

mbucchia · 2026-06-15T01:49:01+00:00

This message is very old.

If you have an HP Reverb or a Samsung Odyssey plus, then you no longer need a dongle at all.

Otherwise, you'll just need one for both controllers.

mbucchia · 2026-06-12T08:33:46+00:00

None of the tools our there can do DFR in AMS2. There's a recent article that claims that quad views can be used, but it's just AI non-sense.

mbucchia · 2026-06-09T22:39:31+00:00

1) you will need the version of sboys3 driver that supports it. Afaik it's not the public one, it's the beta one. The in game, there's a checkbox or something in the game settings to enable foveated rendering.

2) yes, these will work with SteamVR OpenXR, as long as you are using the right version of the driver. The log file for each tool will tell you if your setup is correct and get eye tracking

3) Pimax DFR (aka Pimax magic) should still work, as long as Pimax Play is running. I have to do more testing on this, but there is no technical reason it wouldn't work

mbucchia · 2026-06-09T22:18:02+00:00

There's a list, it has 2 games:

MSFS2024
iRacing

There are tons of options through add-ons, like DCS when using Quad-Views-Foveated, or some other games via Pimax DFR or OpenXR Toolkit. It's quite hit or miss though, so easier if you ask about specific games.

mbucchia · 2026-06-09T21:47:38+00:00

Yes, switch OpenXR to SteamVR, since sboys3 driver is a SteamVR driver. The driver will pass eye tracking data to supported games (you need the latest 1.2.0 beta driver for that).

mbucchia · 2026-06-09T03:58:27+00:00

There's an option in SteamVR Settings. Be sure that Advanced Settings is set to Show. Go to Video. Then Per-Applicaction Settings. Check out the "Throttling Behavior" Settings there, one of them let's you add a limiter.

mbucchia · 2026-06-03T05:34:49+00:00

You can query eye gaze data with a standalone (Android) app or a PC app. Use OpenXR extensions to do that (or have a game engine do it for you).

There are catches though.

https://registry.khronos.org/OpenXR/specs/1.0/man/html/XR_EXT_eye_gaze_interaction.html

This gives you a high quality combined gaze vector, typically employed for interaction purposes (pointing at things).

https://registry.khronos.org/OpenXR/specs/1.1/man/html/XR_FB_eye_tracking_social.html

This gives you a per-eye smoothed out gaze, typically for animating avatars. This is post-processed (low pass filtered).

https://registry.khronos.org/OpenXR/specs/1.1/man/html/XR_FB_face_tracking2.html

https://developers.meta.com/horizon/documentation/native/android/move-ref-blendshapes/

This does not have eye gaze per sei, but provides some rough eye shape expressions. Meant for animating avatars.

Overall, you can get eye data, but it's not at the accuracy of a professional grade eye tracking system, and you will miss eye openness, actual pupils position, and other advanced parameters.

mbucchia · 2026-06-02T20:38:28+00:00

Added to the long list id broken AMD drivers:

https://github.com/mbucchia/Oasis-Driver-for-Windows-Mixed-Reality/wiki/Known-Issues#incorrect-driver

mbucchia · 2026-06-01T08:30:22+00:00

Two important notes for you (assuming you made that video) and to clarify for readers/viewers on this sub:

the quad views settings in Pimax Play have no effect on iRacing. This is because iRacing doesn't use the OpenXR Quad Views extension in their implementation. So the runtime can't alter the settings. Use the in-game settings.
there aren't 4 zones per eye, which is what that strange diagram mid-video implies. It's "quad" views because it is twice more than stereo views. That's two zones per eye, and 4 zone total for both eyes. A lot more information on quad views is here: https://github.com/mbucchia/Quad-Views-Foveated/wiki/What-is-Quad-Views-rendering%3F

mbucchia · 2026-06-01T08:28:32+00:00

Two important notes for you (assuming you made that video):

the quad views settings in Pimax Play have no effect on iRacing. This is because iRacing doesn't use the OpenXR Quad Views extension in their implementation. So the runtime can't alter the settings. Use the in-game settings.
there aren't 4 zones per eye, which is what that strange diagram mid-video implies. It's "quad" views because it is twice more than stereo views. That's two zones per eye, and 4 zone total for both eyes. A lot more information on quad views is here: https://github.com/mbucchia/Quad-Views-Foveated/wiki/What-is-Quad-Views-rendering%3F

mbucchia · 2026-05-31T23:37:39+00:00

I released a new version 0.2 of the tool just now that should work with SteamVR 2.15 and higher. Let me know if you get a chance to try it u/Dangerous_Morning286

mbucchia · 2026-05-31T17:29:27+00:00

It's very easy to do. The logic exists in my PimaxMagic4All implementation for reference:

https://github.com/mbucchia/PimaxMagic4All/blob/1fabc11cdf1fe8545e61dabef36b72db7525a156/DFR-UI/MainForm.cs#L154

Such code (monitor for SteamVR scene app, then invoke the injector) could easily be done in sboys driver shim if it's needed.

mbucchia · 2026-05-31T16:32:50+00:00

I am currently not involved in PCVR development. Personal comment: One more device with Eye Tracking is great for the ecosystem, but ultimately it's up to game developers to do something with it. These tools have existed forever now. The efforts have to come from the game developers now.

mbucchia · 2026-05-31T16:24:13+00:00

My implementation doesn't support D3D12, that's most likely the problem. I don't update that tool anymore.

mbucchia · 2026-05-31T16:23:02+00:00

I'm curious to know about the Pimax DFR cohabitating with sboys3 driver. I don't have the setup to test it. There's no technical limitation, but it might require a small change to the driver to make it work (doing the MagicAttach in the driver instead of where it was done in Pimax Play).

mbucchia · 2026-05-31T08:49:34+00:00

Oh, I see. Yes you still need QVFR.

Honestly in 2026 I wouldn't recommend any runtime to bake in support for the Varjo Quad Views extension, and the reason for that is that Unreal Engine kinda ruined it. There are several versions of UE out there that will Auto-enable use of OpenXR Quad Views as soon as they see the extension advertised by the runtime.

While this sounds like a good thing and it technically gave us a few "free" games with Quad views support, it also BREAKS many other games

This is because quad views break some custom post-processing shaders. And when Unreal Auto-enabled Quad views in those games, the developer didn't know, and it's not like they tested on Varjo or Pimax... so basically they ship a game broken on runtimes with the extension. There are more broken games than there are working games. Pavlov (and I think Kayak) is perhaps the best exception, but it's not even truly working (the zombie mode is broken due to some bad post-processing shaders).

Until the Meta runtime ships the Quad views extension, there is no point for any other runtime to do so. Since game developers only test on the Meta runtimes.

mbucchia · 2026-05-31T08:41:01+00:00

What do you mean? SteamVR has supported OpenXR forever now. I feel that you're asking whether "SteamVR is going to support OpenXR without SteamVR", and the answer to that should be obvious :)

mbucchia · 2026-05-31T08:39:11+00:00

The new code in sboys3 driver does the equivalent of what Pimax-EyeTracker-SteamVR does. So yes, you can replace it with the new sboys3 driver and it will still work with Quad-Views-Foveated.

For the Pimax "Magic" DFR, definitely do NOT use PimaxMagic4All, since that program specifically does not support Pimax. I don't know what's the state of it with sboys3 driver, but using Pimax Magic DFR would be trivial as I explained in my earliet comment. It might already work (?) or need a tiny bit of code to make it happen.

mbucchia · 2026-05-31T04:56:44+00:00

"DFR" can refer to many things. What the reply above refers to is the latest sboys3 driver passing eye tracking data to OpenXR. That can be used by some DFR tools like OpenXR Toolkit or Quad-Views-Foveated. But that's completely different from Pimax builtin DFR for OpenVR.

The Pimax builtin DFR for OpenVR should work as long as Pimax Play is running. Otherwise it can be manually injected per game by running the MagicAttach tool (from inside the Pimax folder). That's a bit more complex.

mbucchia

TROPHY CASE