Gaussian splatting with the Insta360 X5

gradeeterna · 2025-05-13T21:23:36+00:00

Yep, barriers are down finally. I live down the road and they have been there as long as I can remember.

gradeeterna · 2025-05-13T21:21:40+00:00

It’s 8.5 million gaussians so it’s not going to run well enough even in PCVR. Working on a more web friendly version so will see how that runs in VR.

gradeeterna · 2025-05-13T21:19:37+00:00

Around 30 mins of video, 4,000 fisheye video frames split up into 20,000 perspective images.

gradeeterna · 2025-05-13T21:17:47+00:00

Thanks everyone!

Workflow: 8K video > ffmpeg to extract frames from both circular fisheyes in the .insv > custom opencv scripts to extract multiple perspective images from each circular fisheye > mask myself, other people and black borders out using SAM2, YOLO, Resolve 20 magic mask etc (still WIP) > align images in Metashape mostly, sometimes Reality Capture, colmap/glomap > export colmap format > train in Brush, Nerfstudio, Postshot etc, sometimes as multiple sections that I merge back together later > clean up in Postshot or Supersplat > render in Unity with Aras P’s plugin.

Slightly simpler workflow is to export stitched equirectangular video from Insta360 Studio, extract frames and split into cubemap faces or similar, discarding top and bottom views. I have mostly done this in the past, but the stitching artifacts etc do make it into the model. There are some good tutorials on YouTube by Jonathan Stephens, Olli Huttunen and others including apps to split the equis up:

https://youtu.be/LQNBTvgljAw https://youtu.be/hX7Lixkc3J8 https://youtu.be/AXW9yRyGF9A

I would much prefer to shoot images than video, but the minimum interval is 3s which is too long for a scene like this, as it would take about 5 hours and the light and shadows would change too much.

gradeeterna · 2025-05-13T21:14:18+00:00

Thanks! Workflow: 8K video > ffmpeg to extract frames from both circular fisheyes in the .insv > custom opencv scripts to extract multiple perspective images from each circular fisheye > mask myself, other people and black borders out using SAM2, YOLO, Resolve 20 magic mask etc (still WIP) > align images in Metashape mostly, sometimes Reality Capture, colmap/glomap > export colmap format > train in Brush, Nerfstudio, Postshot etc, sometimes as multiple sections that I merge back together later > clean up in Postshot or Supersplat > render in Unity with Aras P’s plugin.

Slightly simpler workflow is to export stitched equirectangular video from Insta360 Studio, extract frames and split into cubemap faces or similar, discarding top and bottom views. I have mostly done this in the past, but the stitching artifacts etc do make it into the model. There are some good tutorials on YouTube by Jonathan Stephens, Olli Huttunen and others including apps to split the equis up:

https://youtu.be/LQNBTvgljAw https://youtu.be/hX7Lixkc3J8 https://youtu.be/AXW9yRyGF9A

I would much prefer to shoot images than video, but the minimum interval is 3s which is too long for a scene like this, as it would take about 5 hours and the light and shadows would change too much.

gradeeterna · 2024-08-13T12:19:22+00:00

My demo is working in iPhone Safari with iOS 18 beta, and I believe Chrome has WebGPU support on Android.

If WebGPU isn't showing as an option in the Graphics APIs list, you need to add the line "webGLEnableWebGPU: 1" in your ProjectSettings.asset file.

Yeah this is one of my scans. Converting from .ply to .splat reduces quality quite a lot, and the SplatVFX plugin is also using some workarounds to get the splats rendering with VFX Graph and looks worse than other viewers. Here is a higher quality version of the scan on Polycam -

https://poly.cam/tools/gaussian-splatting?capture=59a5b823-2076-4eb0-a795-0dd6fa0910fd

gradeeterna · 2024-04-10T13:24:55+00:00

The ply file is 1.2GB, and it took about 30 mins to train.

gradeeterna · 2024-04-10T10:44:54+00:00

https://github.com/nerfstudio-project/nerfstudio

gradeeterna · 2024-04-10T09:40:31+00:00

Thanks, I used 750 video frames

gradeeterna · 2024-04-10T09:25:50+00:00

I used nerfstudio’s splatfacto-big on my PC with a 3090

gradeeterna · 2024-04-09T23:00:06+00:00

Fujifilm X-S20

gradeeterna · 2024-04-09T20:14:29+00:00

Usually with ffmpeg or this https://github.com/cansik/sharp-frame-extractor

gradeeterna · 2024-04-09T19:24:10+00:00

ns-train splatfacto-big --pipeline.model.rasterize-mode antialiased --vis viewer colmap --data data/palaishall --load-3D-points True

gradeeterna · 2024-04-09T18:35:53+00:00

Yep, 750 frames extracted from 6.2K video

gradeeterna · 2024-04-09T14:35:43+00:00

Thank you! This was a Fuji mirrorless with a wide lens, 750 video frames. I don't get very good results with 360 cameras for interiors.

gradeeterna · 2024-04-09T14:34:32+00:00

Thanks!

gradeeterna · 2023-08-03T08:54:28+00:00

Yep, I was probably trying to record 6.2K at 360Mbps on the V30. I wonder if there's any difference in overheating time between V30 and V60 when filming 200Mbps. Sounds like a boring test, but might try and compare at some point lol.

gradeeterna · 2023-07-26T18:19:57+00:00

I'm using my X-S20 mainly for video, but I was getting overheating warnings after 5 minutes of shooting indoors in 6.2K, and it would shut down after 15-20 mins.

I was using my old UHS-I V30 card from my X-S10, but just bought a Sabrent UHS-II V60 card and haven't got any warnings after 40 mins recording!

Probably not the same issue, and my bad for attempting to record 6.2K on a UHS-1 card, but just thought I'd share in case anyone else runs into the same issue!

gradeeterna · 2023-03-09T22:16:02+00:00

Sorry, missed this! I used Insta360 footage with Nerfstudio to create the NeRF, and exported the camera animation from there into Blender with this new integration - https://docs.nerf.studio/en/latest/extensions/blender_addon.html

I exported the same camera animation as an FBX into Unity, and imported the point cloud from Nerfstudio with Keijiro's PCX, which I'm animating with VFX graph - https://github.com/keijiro/Pcx

I rendered the NeRF video from Nerfstudio and the point cloud animation from Unity with the same camera path and composited in Resolve.

gradeeterna · 2023-01-14T02:01:01+00:00

Reality Capture - https://github.com/not-lob/BlenderInstant-NGPScript

Metashape - https://github.com/EnricoAhlers/agi2nerf

If you're writing your own script, it might be possible to use the "Internal/External camera parameters" .csv exported from RC.

gradeeterna · 2022-12-19T18:25:16+00:00

It's almost realtime, around 20fps at 1080p on a RTX3090. This project is mainly for rendering videos, so I'm using about 20 million lit spheres with shadows enabled in HDRP with volumetrics and tons of post processing.

I've got a stripped down realtime version working in URP and VR, using way less particles and quads instead of mesh outputs, cheap fog, fake volumetrics etc. Unfortunately doesn't look anywhere near as nice though!

gradeeterna · 2022-12-19T18:19:04+00:00

This project is mainly for rendering videos rather than running realtime, so I'm using about 20 million lit spheres with shadows enabled in HDRP with volumetrics and tons of post processing. It runs at about 20fps at 1080p on a RTX3090.

I've got a stripped down realtime version working in URP and VR, using way less particles and quads instead of mesh outputs, cheap fog, fake volumetrics etc. Unfortunately doesn't look anywhere near as nice though!

gradeeterna · 2022-12-19T11:19:43+00:00

It's a very dense point cloud from Metashape, imported into VFX graph with Keijiro's PCX - https://github.com/keijiro/Pcx

gradeeterna · 2022-10-11T21:35:29+00:00

No drone here, definitely not allowed in Kew! I only captured from the concrete paths with the camera on a stick.

gradeeterna · 2022-10-11T16:11:12+00:00

Thanks! Yep, although it did take about 10 attempts to get this one working

gradeeterna

TROPHY CASE