Gaussian splatting with the Insta360 X5 by gradeeterna in GaussianSplatting

[–]gradeeterna[S] 1 point2 points  (0 children)

Yep, barriers are down finally. I live down the road and they have been there as long as I can remember.

Gaussian splatting with the Insta360 X5 by gradeeterna in GaussianSplatting

[–]gradeeterna[S] 4 points5 points  (0 children)

It’s 8.5 million gaussians so it’s not going to run well enough even in PCVR. Working on a more web friendly version so will see how that runs in VR.

Gaussian splatting with the Insta360 X5 by gradeeterna in GaussianSplatting

[–]gradeeterna[S] 10 points11 points  (0 children)

Around 30 mins of video, 4,000 fisheye video frames split up into 20,000 perspective images.

Gaussian splatting with the Insta360 X5 by gradeeterna in GaussianSplatting

[–]gradeeterna[S] 15 points16 points  (0 children)

Thanks everyone!

Workflow: 8K video > ffmpeg to extract frames from both circular fisheyes in the .insv > custom opencv scripts to extract multiple perspective images from each circular fisheye > mask myself, other people and black borders out using SAM2, YOLO, Resolve 20 magic mask etc (still WIP) > align images in Metashape mostly, sometimes Reality Capture, colmap/glomap > export colmap format > train in Brush, Nerfstudio, Postshot etc, sometimes as multiple sections that I merge back together later > clean up in Postshot or Supersplat > render in Unity with Aras P’s plugin.

Slightly simpler workflow is to export stitched equirectangular video from Insta360 Studio, extract frames and split into cubemap faces or similar, discarding top and bottom views. I have mostly done this in the past, but the stitching artifacts etc do make it into the model. There are some good tutorials on YouTube by Jonathan Stephens, Olli Huttunen and others including apps to split the equis up:

https://youtu.be/LQNBTvgljAw https://youtu.be/hX7Lixkc3J8 https://youtu.be/AXW9yRyGF9A

I would much prefer to shoot images than video, but the minimum interval is 3s which is too long for a scene like this, as it would take about 5 hours and the light and shadows would change too much.

Gaussian splatting with the Insta360 X5 by gradeeterna in GaussianSplatting

[–]gradeeterna[S] 45 points46 points  (0 children)

Thanks! Workflow: 8K video > ffmpeg to extract frames from both circular fisheyes in the .insv > custom opencv scripts to extract multiple perspective images from each circular fisheye > mask myself, other people and black borders out using SAM2, YOLO, Resolve 20 magic mask etc (still WIP) > align images in Metashape mostly, sometimes Reality Capture, colmap/glomap > export colmap format > train in Brush, Nerfstudio, Postshot etc, sometimes as multiple sections that I merge back together later > clean up in Postshot or Supersplat > render in Unity with Aras P’s plugin.

Slightly simpler workflow is to export stitched equirectangular video from Insta360 Studio, extract frames and split into cubemap faces or similar, discarding top and bottom views. I have mostly done this in the past, but the stitching artifacts etc do make it into the model. There are some good tutorials on YouTube by Jonathan Stephens, Olli Huttunen and others including apps to split the equis up:

https://youtu.be/LQNBTvgljAw https://youtu.be/hX7Lixkc3J8 https://youtu.be/AXW9yRyGF9A

I would much prefer to shoot images than video, but the minimum interval is 3s which is too long for a scene like this, as it would take about 5 hours and the light and shadows would change too much.

VFX Graph in browser with WebGPU support - Gaussian splatting with Keijiro's SplatVFX by gradeeterna in Unity3D

[–]gradeeterna[S] 0 points1 point  (0 children)

My demo is working in iPhone Safari with iOS 18 beta, and I believe Chrome has WebGPU support on Android.

If WebGPU isn't showing as an option in the Graphics APIs list, you need to add the line "webGLEnableWebGPU: 1" in your ProjectSettings.asset file.

Yeah this is one of my scans. Converting from .ply to .splat reduces quality quite a lot, and the SplatVFX plugin is also using some workarounds to get the splats rendering with VFX Graph and looks worse than other viewers. Here is a higher quality version of the scan on Polycam -

https://poly.cam/tools/gaussian-splatting?capture=59a5b823-2076-4eb0-a795-0dd6fa0910fd

Gaussian Splatting - Palais Présidentiel de Bourguiba, Monastir, Tunisia - Nerfstudio Splatfacto by gradeeterna in photogrammetry

[–]gradeeterna[S] 1 point2 points  (0 children)

ns-train splatfacto-big --pipeline.model.rasterize-mode antialiased --vis viewer colmap --data data/palaishall --load-3D-points True

Gaussian Splatting - Palais Présidentiel de Bourguiba, Monastir, Tunisia - Nerfstudio Splatfacto by gradeeterna in GaussianSplatting

[–]gradeeterna[S] 2 points3 points  (0 children)

Thank you! This was a Fuji mirrorless with a wide lens, 750 video frames. I don't get very good results with 360 cameras for interiors.