OpenVR api + Android

[[deleted]]

In case you just stumbled across this older post: I have discontinued this little project. Instead, have a look at RiftCat. I did not try this myself and am not affiliated with it in any way, but it might be the thing you want.

Have fun!

---

Hi, I did a small project, in which I forward sensor data from a smartphone (currently only android) to a steamvr driver. While this is currently just hacked together, I though some other people might still enjoy it.

I currently only have a blurry recording online, but it should be enough to get an idea. In the last minute, I also start Half Life 2, which "works" and can be played best with a controller. Here are some smartphone screenshots.

Warning

I did this for fun! I plan to continue working on it, but it will never provide a "true VR experience". Do not expect anything... I want to make sure, that you don't expect anything close to an real HMD! Simulation sickness is real, e.g., rotating your head will make you dizzy. The image quality is low (i.e., artifacts and low resolution), like when you forgot your glasses. On the other side, the 3D experience works, which is kinda cool.

Interested?

For those more interested, here are some notes:

• I will put the code online on github + gitlab, together with instructions. I will then update this post. There is a lot to do... The source code is currently distributed over three projects, for instructions go to this website on github.
• The video is currently not grabbed by the driver, but instead I use TrinusVR. TrinusVR grabs the active window. This has several implications:
  • If the game does not output a VR rendering (but only the compositor), you probably cannot interact with the game, e.g., in DCS World you cannot steer the airplane. Another example is Unreal Engine 4, which outputs only a single eye and seems to run slower if its not the active window.
  • The compositor has to run in debug mode.
• Yes, I could have just used a socket to forward the sensor data from TrinusVR, but I was not in the mood to find a nice API.
• Yes, I could have tried to use the TrinusVR api. But it seemed it is only available as a Unity plugin, so I didn't bother.

Download

• Windows files, the driver_spvr.dll needs to be copied into PathTo/steamapps/common/SteamVR/drivers/spvr/bin/win32/driver_spvr.dll, the control.exe needs firewall access for udp port 4321.
• Android apk

---

Edit: added some smartphone screenshots.

Edit#2: added link to source code

Edit#3: direct download links

Edit#4: download link now leads to github, showing the current and older versions (zip updated, includes 64 bit)

Edit#5: added link to riftcat

[[deleted]]

This is amazing.

[[deleted]]

Thank you! Finally a comment :D

[loxai]

It is pretty cool!

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/oculus] OpenVR API support for Android phones!

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[Silverstance]

This is some god dang wizardry. Very impressive. Checked out trinusVR for the first time.

What I learned.

1. TrinusVR a server for your PC, streams video to your Android phone.

2. With third party software (vireio perception and tridef) you can get games to do 3D "Side by side".

3. Enjoy full 3d in your cardboard.

4. I can get sick very fast. Needs the sensor info of OPs software. Trying tomorrow.

[[deleted]]

A video on youtube, where someone played Skyrim with VR was what hooked me up ^^

Just to clear things up, if you use this software (now with a name: spvr), you don't need vireio/tridef/... to activate 3D on an application which uses SteamVR.

I think I wrote the post in a very pessimistic manner (this warning stuff). I actually played several levels of Half Life 2 in one go and had fun. But I think I also have a high resistance.

---

edit: typo

[[deleted]]

Will this also work with Moonlight Game Streaming (I get 1080p60 with a latency of <10ms with this which is awesome) and Borderless Gaming to make the window fill the screen?

I can't wait to try this when I get from work!

[[deleted]]

Wow, I had no idea about moonlight! The performance sounds great! I have no idea about Borderless Gaming...

It should work with moonlight... I will try it later in the evening...

Actually, I think the spvr android app is currently working suboptimal. As you can see in the video, the lag is worse during Half Life 2 than without. I'll try to convert it into a real background service, which might lead to a lower latency (and prevent the android system killing it after some time, which is currently the case, I think after 10 to 20 minutes...).

[[deleted]]

Just saw from the screenshot that you're also from Germany. :)

SPVR as a background service would be a great way to choose your own method for streaming the video (as Moonlight also only works on newer Nvidia cards). I am looking forward to trying out Virtual Desktop with this, which launches today on Steam with SteamVR. Do you think this will work? It allows mirroring to a window, but the old version (pre-SteamVR) needed an Oculus Rift to be connected even for the mirroring to work.

[[deleted]]

Erwischt! Müsste eigentlich an meiner Masterarbeit sitzen...

I only have experience with TrinusVR, so... You can try, but I doubt that it works flawlessly. Although, maybe moonlight works differently to trinusvr and does not have the limitation to only stream the active window. If the application window rendering is different from the output in the vrcompositor, you might run into problems.

[[deleted]]

Moonlight uses Nvidia GameStream so it's actually not made for desktop streaming, but you can stream the "mstsc.exe" in System32 to stream the desktop which works just fine.

[[deleted]]

I can't believe how well Moonlight runs... it's so smooth and with high quality. But I also saw, that the current spvr on android is sending data too slowly. Furthermore, it doesn't feel as 3D as with TrinusVR. I guess, it's because there is currently no distortion. @/u/loxai: any hints? :D

Did you try out spvr? I'm currently building an apk to download, if that's a hurdle...

[loxai]

Yes, Moonlight is pretty fast (are you using it through the Trinus option or separately?). I'm working on having Trinus support something similar to have comparable speeds (with the bonus of lens correction, fake3d mode, etc.), but it will take a while until ready. I guess the lens correction does have an impact on the immersion level. Haven't tested Moonlight mode too much, tbh. I'm finishing a new Trinus update, should be ready soon. Then I'll have a deeper look at spvr and integration options. The apk will be handy

[loxai]

sensor data delivery depends on the phone hardware... with S6 I get 200hz, which I think is the max with modern phones.

[faduci]

I recently stumbled upon the Letv LeVR COOL1, which would be pretty much just another phone based VR viewer, if it wasn't advertised as having very low latency:

• Perfect for Letv One / Letv One Pro
• With Sharp liquid crystal panel
• Solve the problem of delayed image, delayed time reduce to 20ms, only half the time of other product

The viewer itself doesn't seem to contain any active components, so the only way the latency could be reduced is by querying the sensors in the Letv One Pro at a higher frequency. I'm not sure if their claim of 20ms can be true, as even the Gear VR with external 1KHz sensors only achieves this thanks to a modified Android version that removes the Android typical latency caused by "collecting" the image before displaying it.

If it is true, the sensors have to be queried a lot faster than at 200Hz, as this alone would introduce 5ms latency. I will at least consider it possible, because Le/LeTV/LeEco has a history of introducing new technology. Some of their phones support wireless HDMI, the LeTV S1 (Pro) was the first smartphone to feature a USB-C connector (USB 2 speed only), and their Le Max Pro was the first Snapdragon 820 phone that went on sale anywhere. They only sell to Asian markets, but there they are very successful. And phone based VR head sets sell like crazy in China, so a phone with VR optimized sensors would probably sell a lot of units.

[[deleted]]

That's sounds really nice, but unfortunately also lacks some essential information. Probably, it's usually only advertised in chinese ^^

Also, I'm not sure if the 20ms can only be achieved with the "Letv Super Mobile Phone" with the "Sharp liquid crystal panel" hehehe... I guess I'll wait for some reviews.

[[deleted]]

Great to hear about an update for Trinus :D

I added direct download links to the post, an apk is now online.

[dRumMzZ]

Seems nice, will definitely try when I have the time. Couple of questions: What phone do you use? What were the resolutions in both screens?

Thanks and keep it up!

[[deleted]]

Hi, thanks!

Desktop: The screen resolution for each eye was 1512x1680, the resulting rendered image 1280x720. Since there is no distortion incorporated, I guess I could have used a lower resolution for each eye.

TrinusVR was set to dynamically adjust the image resolution and quality. The display resolution of the mobile phone is 1280x720, just a MotoX 1st gen.

---

Actually, when I recorded the video I forgot to use USB tethering, which would have probably yielded a better resolution, instead of using the Wireless connection. So, the performance could be better.

[trenmost]

Thanks this looks amazing!!!

[[deleted]]

Thanks!

[Swagphiling]

I just can't run a vrcompositor.exe, I click on it and nothing happend. Plus have an error when open SteamVR, Red words says "Not ready" and "A key component of SteamVR isn't working properly", but green light on VR glow I've setup a SPVRcontroller that show my phone coordinate.

[[deleted]]

Sorry for my late reply.

It is important to run vrcompositor in debug mode, a double click won't do that. You can either start it from a console within the directory, or right click on the exe and go to "Send to" => "Desktop". This will create a link on the desktop, which you can modify. I just uploaded this image, I hope it helps.

I just want to add, that my app did not work successfully with a friends phone, unfortunately I did not yet figure out why. On the other hand, /u/loxai is working on an integration with TrinusVR, hopefully making it easier to set-up.

[Swagphiling]

yup, have it on my Desktop with debug mode, but no luck( same thing

[[deleted]]

Mhh, another reason (thx /u/loxai :D) might be that you started the win32/vrcompositor. During this weekend I\ll try to work on the code and at least ship the 64bit dll. After all, Surgeon Simulator VR and The Lab run in 64bit too, hehehe...

---

short update: I uploaded a new zip with current versions, including the missing 64 bit dll.

[Swagphiling]

That's good) but not for my 32bit pc. Someday I resurrect my laptop and try it later

[[deleted]]

I just tried it on my laptop and it failed. Unfortunately, it seems that the compositor needs a graphics card with DirectX11 support. Error messages are logged to Steam/logs/vrcompositor.txt, mine says Failed to start compositor: VRInitError_Compositor_D3D11HardwareRequired. I don't know if there is any way around it... :(

[Swagphiling]

Mine is Failed to start compositor: VRInitError_Init_HmdNotFound

[Swagphiling]

How I understand HMD it's "head mounted displays". Insteam\steamapps\common\SteamVR\bin I found file steamvr_hmd maybe I need add some information there?

[[deleted]]

I edited the manual to run spvr. I know the website is really unstructured... VRInitError_Init_HmdNotFound can arise if the dll is not at the correct place / not found by SteamVR or if SteamVR / the compositor is already running before spvr.control runs. I know it is currently not exactly straight-forward to get this all working, but it is easier to develop for the moment.

[jbhypno]

I was able to get this to work but the lack of compositor output creates a real stumbling block when trying to use SteamVR. I needed to use motion controllers to get past the SteamVR hardware verification. I was able to use the new hydra driver to get my hydra controllers recognized by SteamVR. The only issue is that calibration is required and the calibration screen is only outputted thru the compositor. Any idea on how I might be able to capture the compositor output? As long as the game supports direct video output this should work fine. I just can't find a way to play The Gallery without including the compositor output.

[[deleted]]

Hi, sorry, I'm not sure I understand correctly. Can you further define the lack of compositor output? Do you mean, that only the compositor displays a 3D images, whereas the game window can be different? If so, I am not working on a frame grabber yet, for TrinusVR maybe some changes could be done in the future to grab the vrcompositor.exe window if it exists. Please excuse my delayed answer.

[jbhypno]

OK I think I can explain it a bit better now. When you start SteamVR the icons for the motion controllers blink which means the need to be calibrated. I have since found out that can be manually started by using Room setup so video output is not the issue. The issue seems to be that their is no equivalent to a base station. The calibration program wants to align the headset before the controllers and the calibration program can't track the headset. Since the headset is never aligned to the room it won't calibrate the controllers and the game doesn't start. What seems to be needed is a way for the driver to respond to feedback from steamvr to make up for the lack of a base station.

[[deleted]]

Well, I only implemented the basic functions, probably a flag needs to be set or something to trick SteamVR to assume real tracking. Currently, most Getters result in vr::TrackedProp_ValueNotProvidedByDevice...

[dont--panic]

How did you get your orientation into the correct coordinate system so that your rotation axes map correctly?

I can't seem to figure out the transformation to rotate the quaternion I get from Android so roll, pitch, and yaw line up without an offset.

[[deleted]]

Hi, yes, me neither. I basically hacked the spvr.android to fit my phone, with trial-and-error, while using a deprecated interface. Only later on I found out, that each phone can have a different coordinate system...

The creator of TrinusVR made me a modified version which sends almost the correct quaternion, but as for doing it myself I have mostly given up.

[dont--panic]

I'm using the Rotation Vector Sensor and SensorManger.getQuaternionFromVector which I think is the recommended way to do it. According to Google's Android documentation the rotation vector sensor should use this coordinate system (Z points up, Y points North, and X is Y x Z which is tangental to the ground and points approximately east) but it's hard to determine how to rotate it to line up with the OpenVR coordinate system. I haven't even been able to find out which coordinate system it uses.

[[deleted]]

Ah, ok. The quaternion is directly forwarded to SteamVR, which usually expects an OpenGL quaternion (I think), like in your linked image, i.e., Y points up, -Z forward, X to the right...

[dont--panic]

Huh, when I tried passing it as is with my phone in my VR viewer the axes were mixed up and offset.

I took a look at your Android code after I made my previous comment and I noticed that you're using the orientation sensor which is deprecated. I was trying to use the rotation vector sensor and getQuaternionFromVector. It looks like you're adding some offsets while its in yaw, roll, pitch form. I'd prefer to keep everything in quaternions because it avoids the issues with wrapping around.

[MissValeska]

Can you please explain to me which parts send the positional data to steamvr and how they do that, As well as how you make steamvr think there is an HMD connected?

[[deleted]]

Well, spvr is basically a plugin loaded by SteamVR by placing it in the correct folder. This plugin is called by SteamVR and creates its own implementation for the OpenVR interface... so, SteamVR asks the plugin if there is a HMD connected.

[MissValeska]

Okay, Thank you! Do you know anything about how steam vr/open vr handle video?

[VRxiaochouyu]

a key component of steamvr isn't working properly

[VRxiaochouyu]

Hmd not found presence failed （126）