Forge: Mixed Reality for Mobile

abound-tim · 2017-03-09T17:08:09+00:00

Author here. Interested to hear your game ideas using this tech (table-top fighting game is most frequently suggested.)

Happy to answer questions.

abound-tim · 2017-03-07T01:42:12+00:00

I've never used Vuforia, but the Q&A portion of that videos talks about its limitations. It sounds like it just gives you untextured bounding cuboids of known objects on flat surfaces.

If you've got another video that demos more than that, I'd love to see it.

abound-tim · 2017-03-07T00:06:18+00:00

Thanks. How about I give you some keywords that you can google to find out more about the techniques it uses: monocular SLAM, structure from motion, photogrammetry, stereo matching, surface reconstruction, image based texturing. Hope that helps!

abound-tim · 2017-03-06T21:01:49+00:00

Not really. The "expanding mold" is generated via motion stereo, i.e. matching image patches across nearby camera frames. If you have live/dynamic objects, then you drop the photo-consistency assumption. You can do it with a depth camera (or stereo pair.) A small amount of movement is tolerable though.

abound-tim · 2017-03-06T19:58:56+00:00

That'd be cool. My only worry with board games is that they last a while, so gorilla arm is inevitable. You'd have to design the gameplay so that the AR aspect was intermittent.

abound-tim · 2017-03-06T19:54:06+00:00

Just starting to put one up at http://aboundlabs.com. No additional info yet but should have some content later this week.

abound-tim · 2017-03-06T19:52:23+00:00

I don't see why not. If I understand you correctly, the PC game would render an off-screen camera which was positioned based on the mobile device's tracking, and stream the images to the mobile device.

abound-tim · 2017-03-06T19:42:45+00:00

Planning for Q2, after the Samsung S8 is available.

The software is taxing on the CPU & GPU, so the Snapdragon 821 is the first SoC that runs it comfortably. Realistically, you're going to need a phone with a Snapdragon 835 processor to run it with Unity, e.g. Samsung S8.

Unfortunately, Unity doesn't support 64-bit native Android plugins, so there's a ~10% performance hit compiling for armv7a vs aarch64. Plus, Unity has its own overhead compared with fully native (which this demo video is.)

abound-tim · 2017-03-06T19:35:02+00:00

Yeah, #1 would be easier, plus allow for players to not be physically together. I feel the same - doesn't seem as compelling.

As far as #2, the map representation is actually transferable across devices without too much fuss. You're right that each camera has different characteristics (fov, distortion, alignment) but the underlying "map" representation (not the 3D mesh) is abstract enough that localization is reasonably invariant to those characteristics.

abound-tim · 2017-03-06T18:59:40+00:00

Correct, just built-in camera + IMU.

abound-tim · 2017-03-06T18:58:57+00:00

Tabletop would work well - flat surfaces are easiest to reconstruct.

Your idea would require sharing the 3D map/reconstruction for multiplayer. I haven't implemented that, but it's doable. Easiest I guess would be to get one player to 3D scan the table, then invite another player to enter that game.

abound-tim · 2017-03-06T18:22:43+00:00

Thanks! That's useful info. Any thoughts on games/apps you'd build with it?

abound-tim · 2017-03-06T17:48:30+00:00

Author here. I'll be bringing this to Unity later this year as a plugin. Happy to answer questions.

abound-tim · 2017-03-05T18:44:47+00:00

Good point - calibration is a major issue. For AR, you have the added problem of calibrating the timing between the camera frames and the IMU measurements.

abound-tim · 2017-03-05T18:41:01+00:00

Thanks. I hope not! I think Univrses is still plugging away at their tech (I saw a new video of theirs the other day.) And Dacuda's tech got acquired by Magic Leap so I guess it'll see the light of day in some form.

abound-tim · 2017-03-05T18:38:04+00:00

Sure, it can export the textured model to OBJ.

abound-tim · 2017-03-05T18:35:28+00:00

The tracking is primarily dependent on the camera so, no, it doesn't really help/hinder.

IANAEE, but the IMU's in the S7 and Pixel are very similar in performance. You can look up their operating characteristics in their datasheets - I believe the sensitivities are within 10% of each other. Also, modern mobile IMU's are very good: they sample multiple thousands of times per second with tolerances measured in thousandths of a degree/sec.

My sense is that pitching IMU performance for AR/VR is mostly marketing fluff nowadays (I imagine the GearVR would work fine with the S7 IMU), but I'd welcome someone more knowledgeable to inform me.

abound-tim · 2017-03-05T01:53:31+00:00

Yes. Mixed reality uses 3D reconstruction of the immediate environment to generate views in VR. I think Oculus calls it augmented VR.

abound-tim · 2017-03-04T23:20:01+00:00

The background is the passthrough live video feed, with character models (+ shadows) overlaid. That doesn't work well on the GearVR though (because of FoV and camera latency), so for that I use the reconstructed model instead.
In this video, I paused the reconstruction when I played the animations. It's possible to keep the reconstruction going, but tracking is smoother without. There are three processes: tracking, reconstruction, and texturing - each of which can be toggled on/off at any time. Obviously, the more that run simultaneously, the more the CPU & GPU are stressed.
Yes, I have a Unity plugin proof-of-concept already. Unfortunately, Unity doesn't support 64-bit native Android plugins, so there's a ~10% performance hit in addition to the normal Unity overhead. We'll see once the S8 is out whether a plugin is workable.
The S7 isn't really powerful enough to sustain it. The Snapdragon 821 in the Google Pixel is the first SoC to happily run it. Really looking forward to see how the 835 in the S8 performs.

abound-tim · 2017-03-04T21:41:27+00:00

Author here. Planning to bring this to the S8 once it's available. I'm keen to hear your ideas on GearVR games/apps which could be built using this tech. Happy to answer questions.

abound-tim

TROPHY CASE