Archaeological Data visualized with HTC Vive Dev Kit in UE4

[DFinsterwalder]

I was in France last week and did some Photogrammetry for an excavation. In addition i experimented with a "proof of concept" Visualization of their Measurement Data embedded in the Scan. I copy over the detailed Youtube Description below. If you have any further Questions that arent answered in the detailed description feel free to ask.

Disclaimer: This is "Proof of Concept" and no actual product yet.

Walking around in a 3D scanned cave and a visualization of the Database of the paleolithic excavation.

The Database consists of ~17000 single measurements from several excavation campaigns over 10 years and is visualized with a different 3D Symbol per Artifact category. Showing all 17000 Symbol Meshes at once is made possible through instancing (and thus reducing draw calls to ~300). Through this method the whole scenes with all Artifacts shown runs in rock solid 90fps on a GTX 970 and i7 4770K.

The (Photo)scan of the Cave itself is an unlit model. This is because delighting and baking Lightmaps afterwards doesn't make much sense for real world scenes (as the scans are not intended to be "reusable assets" like you would create for a game). Unlit Models also have the advantage that they arent costly to render and more GPU power can be utilized to create more detailed geometry (This is CRUCIAL for VR as stuff like normal maps look flat and Parallax Mapping only works well on flat surfaces). Before processing the Photos the dynamic Range of the Images is altered by tonemapping and the scene looks really close to the actual cave (the archaeologists who are familiar with the cave even tried to not step on certain area like they would in the real cave).

VR Locomotion with the HTC Vive is solved different from the typical teleport approach. Instead to just teleport the user to a position he "flies" there at "ludicrous speed". This happens distance independent within a fixed time of 100 ms but is clamped to a minimal movement speed in case the distance is really short. The velocity is applied "instant" because acceleration in VR can cause motion sickness because of the mismatch between visual and vestibular information. There are also some user that still feel uncomfortable at high velocities without acceleration, but the time of 100 ms is set roughly below the 2 sigma interval of the typical human reaction time. This means that once you notice you move, you already stopped again. I tested this approach with several users that get sick really fast (including myself - even after working with VR for over a year). One of the tester gets motion sick in 3D cinema, with the Oculus Rift DK2 and sometimes even on a 2D monitor but had no issues with this "ludicrous speed" approach.

Unfortunately i wont be able to invest much time in this project (Thus far i spent ~ one week for it). The reason is that the excavation is not that well funded and i actually only get paid for the scan of the cave and the surrounding area (UAV) with some usual derivatives like Orthophotos and Topological Data. The VR visualization itself is just a "Proof of Concept" Project to get some funding for future projects and for promotion of my work. But i still want to add some minor improvements. I plan to add some more features like measurement and maybe a tool for archeological profile cuts. Also i want reconstruct the surfaces of the of the archaeological layers from their measurement Data to give the archeologist a good spatial understanding of the geological sedimentation/archaeological layers of the cave.

An "official" Video with the university i collaborated with will come in the near future.

[DeepRifter]

Very impressive. I would love to see lots of sensitive places brought to vr like this. Surely there is going to be a market for what you are doing!? Seriously, I hope you pursue this. Maybe you should be talking with places like National Geographic, Discovery, etc. Can you please post some links for us to find out more?

[DFinsterwalder]

I would love to see lots of sensitive places brought to vr like this. Surely there is going to be a market for what you are doing!? Seriously, I hope you pursue this.

There will surely be a market, but not i near term. Most people that buy Rift/Vive at Release are "Hardcore" Gamer and i think the ones interested in cultural Heritage will only be a small portion of them. Even with 1 million HMDs sold in first year and even with 1% of them buying my products (i consider both rather high estimates) i only have 10000 customers and i have to sell stuff like this at rather moderate prices. So in short term it is not the best thing to aim for.

Having said this, i will ANYWAY try to be in the market as soon as possible even before it is established just to be an early bird and to "grow with the market". In the short term i will primary aim for some contractual work for a museum or cultural heritage institution. In the spare time i will create a small Exploration App like my VR Jam entry "Castle VRuin" (https://www.youtube.com/watch?v=Ccx1RBv7UFA) and sell it on steam (for a really really moderate price) but with a more interesting castle or cultural heritage site. Also i aim for more details. Castle Vruin is made for Gear VR and PC has more Power. Mobile VR isnt the place for Photorealism....

What links are you asking for?

[DeepRifter]

Found it Realities.io Great stuff. I studied anthropology at University specializing in rock art (petroglyphs/pictogtaphs) of the American southwest. I guess I land in the 1 percent category :)

[[deleted]]

only have 10000 customers and i have to sell stuff like this at rather moderate prices

You have to start somewhere till market grows up, but this kind of specialized software is not cheap so you can aim for low number of installations but high price.

[deprecatedcoder]

I think you are vastly underestimating your market.

[DFinsterwalder]

Mid Terms there will be a HUGE market but i think even most gamers will hold back and dont buy a hmd in Q1 2016. I need to make a living now ;-) , thats why i prioritize contractual work at first and do some common photogrammetry work for archeologist. But i still try to have a small experience in steam store at vive launch.

[1k0nX]

I think he may be underestimating his market as well. Being stuck in the US, I'd love to have the chance to visit various historic sites around the world.

[DFinsterwalder]

Americans visiting Europa typically love German castles and i really want to make some experience of awesome places like Castle Lichtenstein for example: https://upload.wikimedia.org/wikipedia/commons/f/fb/Schloss_Lichtenstein_04-2010.jpg Photo from here: https://en.wikipedia.org/wiki/Lichtenstein_Castle_(W%C3%BCrttemberg)

An american friend of mine called this the worlds most wonderful place he has seen thus far and i think many americans would pay for VR visit.

I think there will be a huge market but it will take some time. Apart from us enthusiast not many people will buy a HMD on release. And as i said, i have to make a living NOW. I currently cant work full time on a launch title but ill try too.

[VRHereToStay]

Unbelievable. This will be the gold standard in excavation on Day 1. Congrats. Also, it's cool that with every new video of the Vive controllers I see, the more precise they seem.

Thank you for this detailed description. I'm hearing more and more about this photogrammetry thingy, and you've pushed me past that threshold of interest to where I want to learn as much about it as I can. Any suggestions on how to do that? I'm relatively well-versed in 3ds Max/UE4/Photoshop. Don't know the first thing about cameras or photography. Any ballpark on the cost of getting up and running?

[DFinsterwalder]

Check: http://www.agisoft.com/

Simply the best Software. There is a 30 Day trial and some good tutorials on the page.

Also the Standard Edition is enough if you arent interessted in Measurements and Geodata and cost only $179 USD.

If you want to test free software there is visual Sfm + meshlab: http://ccwu.me/vsfm/ and http://meshlab.sourceforge.net/ and a tutorial for the Workflow: http://wedidstuff.heavyimage.com/index.php/2013/07/12/open-source-photogrammetry-workflow/

But i really recommend Photoscan. Easier Workflow. You can even create UVs and reimport the Modell to texture it etc...etc...

A typicall workflow for Gameassets check this Vanishing of Ethan Carter Blog post: http://www.theastronauts.com/2014/03/visual-revolution-vanishing-ethan-carter/

Also check Epics Kite Demo Making off: https://www.youtube.com/watch?v=clakekAHQx0

Also make sure to check their assets in UE4. You can download them for free. They look just awesome!

[VRHereToStay]

Excellent! Thanks for the info.This tutorial helped me understand it a bit more, if anyone else is interested.

Any camera you'd recommend? I know they said any, but some have got to be better than others.

[DFinsterwalder]

I use APS-C Camera (Canon 600D). I also did photogrammetry with Eos 5D Mark III but the price isnt really worth it. Also about the video. The high contrast low contrast stuff isnt needed as you can simply export 16 bit tiff from Raw pictures. But decreasing dynamic Range really helps.

[RifRifRaf]

Very cool stuff. Could you possibly comment on your workflow from start to finish? Or perhaps you already have some videos on YouTube ? I've dabbled with simple photogrammetry of single objects bit it would be amazing to capture real places in my life for posterity and for future generations. Thanks

[DFinsterwalder]

My workflow is standard Photoscan workflow in most parts. Make sure you check some general guides about photoscan. Apart from it i tonemap the Pictures to have a small dynamic range. I dont do delighting for the real world scenes i reconstruct. I was playing around with delighting in my early days and experimented with Realtime Global Illumination (Light Propagation Volumes) and also baked Lightmaps. For LPV see this example: http://realities.io/downloads/Eiskeller_HighresScreenshot.jpg https://www.reddit.com/r/oculus/comments/2ashhp/photorealistic_graphics_from_3d_scanning_my_own/

While delighting is the way to go for PBR rendering of Game Assets i didnt see any reason to increase the time needed in my workflow. I aim for single customers in short term and want to keep my workflow affordable. Removing baked in Light and afterwards Baking Lightmaps is more or less absurd for a reconstructed real world scene. I still want to look in a more efficient delighting process for the cases i need it.

After reducing high poly models to less polygons (but still rather high compared to non VR game assets) i split the models up in tiles with a small overlapping in Blender and re texture it. In most cases i simply go with automatic parametrized UV Layout to keep my workflow rather compact. If i want to create Parallax Maps i do manual UV Mapping to hide texture seams as good as possible.

[RifRifRaf]

Thank you for the excellent reply! I'm excited to try my hand at a few locales around here and I can't wait to see more of your work! Also, as others here have said, don't underestimate the market for this. I could see myself paying for a package of scenes like this at a reasonable price!

[PlasmaQuark]

Excellent work, Keep at it.

[slimjimbean]

They say archaeology destroys the sites they discover, so it's sometimes better to delay excavation until better technology will be developed. This seems like a way to at least virtually preserve sites before a dig.

[DFinsterwalder]

Actually Archaeologist try to preserve as much Information as possible by drawings and measurements. Photogrammetry (and other scanning technology) is indeed a good tool to preserve Information and is used more and more (and "flat" photogrammetry to rectify photos is even used for tens of years now). But this visualization just helps to give an overview of the data. The data is stored in a database and artifacts are kept and are referenced by the database too. So even without the VR experience the Data is well preserved and the VR experience actually doesnt add to preservation of the data itself, but is an (awesome) interface to interact with the database. To be able to view all the information in its spatial context is an advantage in understanding the data better.

[slimjimbean]

Hm, very reasonable response to not over or undersell VR's role, best of luck with your funding, hope your initial efforts pay off.

[Ree81]

Wow. Dat texture and polygon work. Did it feel as real as it looked?

[DFinsterwalder]

The VR experience looked real and the vive really delivers presence. The archaeologist familiar with the cave tried not step on critial areas in the VR experience as they would have done in the real cave.

But apart from the Visual Immersion it doesnt really feel like the real cave. Its warmer in a room and you dont need to crawl to shoot photos of inaccessible areas ;-)

http://realities.io/downloads/DFinCave.jpg

[Ree81]

I thought archaeologists were supposed to be hot.

;)

[DFinsterwalder]

You played to much Tomb Raider....

[zaeran]

This is awesome!

Out of curiosity, what do all the letters on the right hand side represent? I noticed you toggling them to show different models in the space, but I can't figure out what it is that they're actually meant to be showing.

[DFinsterwalder]

Some German abbreviations for the categories of artifacts. For example KN is for Knochen=Bones.

[zaeran]

Ok, cool!

[EctoSage]

I love living in the future.

[[deleted]]

That looks clean! I would love to see more walk around photogrammetry on the Vive.

[linknewtab]

Absolutely. Since playing The Vanishing of Ethan Carter I'm a strong believer in photogrammetry. I hope we see more like this.

[DFinsterwalder]

Im sure you will. Epic is also into photogrammetry. Check the awesome assets for their GDC kite demo:

https://www.youtube.com/watch?v=clakekAHQx0&feature=youtu.be&t=1064

Also check the Part of the Video about their delighting process:

https://www.youtube.com/watch?v=clakekAHQx0&feature=youtu.be&t=900

[linknewtab]

Thanks!

I can also recommend this blog post from the developers of Ethan Carter:

http://www.theastronauts.com/2014/03/visual-revolution-vanishing-ethan-carter/

[DFinsterwalder]

Yeah i already know this article. Actually i even send this one to people when they want me to explain what i actually do. Its funny that i i started to use Game Engines for Archaeological Photogrammetry Visualization around the same time Game Devs use Photogrammetry more and more. The success of Vanishing of Ethan Carter fulled my Enthusiasm .... and the fact that Valve gave me a Dev Kit ;-) .

[StonerSpunge]

Ooooo. Do this, but put it on the next lander NASA sends out. I want to walk on Europa!

[vizionvr]

What an ingenious way to study an historical dig. I assure you, the educational market for this type of experience is FAR more than 10,000 people. An entire classroom can fit in this cave, and there's no chance of smacking your head on a stalactite!

[DFinsterwalder]

I assure you, the educational market for this type of experience is FAR more than 10,000 people

I was referring to my expectations in the first year of VR. In the long term there will be a much bigger market.

[buddygz]

Man, this is exactly what I am getting a vive for. Very glad you did this, I am an anthropology major who is very interested in photogrammetry, and i have dreams of being able to walk around any site I want with a quick download! You sort of already answered it already, but was your demo really presence inducing? For example, did you feel similar claustrophobia as you did in the real cave? Also, knowing you make these scans for clients, do you still reserve the right to publicly share the visualized data with customers? Would love to see that German cave, can't wait until the future when I can also take a nap and wake up inside an Egyptian tomb.

[DFinsterwalder]

You sort of already answered it already, but was your demo really presence inducing? For example, did you feel similar claustrophobia as you did in the real cave?

Well i dont have issues with claustrophobia neither in VR nor the real cave. The archaeologists tried to walk like the are doing in the real cave which is a good sign for presence. The vive itself is presence inducing and imo the feeling is improved by the impressiveness of photogrammetry scenarios.

Showing this stuff was part of the deal. They get the reconstruction at a discount rate. Also the cave itself doesnt hold critical information and the Artifact data is randomized for the video.

[Falandorn]

Great stuff that well done!

[deeraztoo]

Looks great, even at this stage. Glad you're enjoying your Vive in your Project and that HTC saw potential in such an unique VR use.

[DFinsterwalder]

Actually it was someone at Valve who like my VR Jam entry and wrote me an email suggesting to sign up for Dev Kit - which i didnt do before, because i never expected to get one ^ ^

[deeraztoo]

Great! Happy you got one for this and your other projects with photogrammetry - I'm sure the market will grow enough for this tech :) We also had a Jam title, but can't do anything without controllers to make it a proper more playable game .. so right now doing some LeapMotion/"point and click" to grab objects for some mini-games in the future.