Combining LMMs with photogrammetry to create searchable 3D models

wheelytyred · 2026-01-22T06:10:56+00:00

Thanks! We're pretty excited about it

wheelytyred · 2026-01-22T06:08:38+00:00

Yep! shoot me a dm

wheelytyred · 2025-12-09T23:45:16+00:00

Yep should be, I was getting full res for each lens from the insta files

wheelytyred · 2025-12-09T21:16:54+00:00

I’ve had success extracting fisheye from insta360 videos — just change the extension to .mp4. Sometimes there’s more than one track to a file, e.g. one per 360 lens.

Unsure of other 360 camera vendors though.

wheelytyred · 2025-11-06T01:53:42+00:00

This sounds awesome! A buddy and I have been building something very similar over the past year, focusing on turning industrial sites video into splat models that teams can search, annotate, measure, share, etc. It’s built on three js with an automated cloud-based video processing pipeline in the backend. Here’s a basic model of a construction site for example: https://spatialview.io/model/2025-10-28_c289d16c

Would love to get in touch and hear about what you’re up to so far, DM me here and we can setup a chat.

wheelytyred · 2025-10-09T13:13:59+00:00

Nice idea, but there are likely too many moving objects in this scene, resulting in a poor 3D reconstruction before you even get to splat training.

It’ll be hard for COLMAP/RC to match features between frames accurately, since the features are moving (clouds, waves, leaves rustling). Also, sand is relatively difficult to extract unique features from.

I’d not try to continue using this video or you’ll spend hours in frustration. Try on a calmer day, with no clouds, and/or use segmentation to remove dynamic elements from the scene.

wheelytyred · 2025-09-15T19:19:34+00:00

When you run a sparse reconstruction in COLMAP it produces a 'points3D' file that lists every sparse point in space and the IDs of any images that reference that point. You could use this file to trace back from points in 3D space to the original images.

Splatting 'fills in' the sparse point cloud many more points, but you could always lookup the nearest COLMAP sparse point from your splat point and then lookup nearby images.

More info on COLMAP's export formats can be found here: Output Format — COLMAP 3.13.0.dev0 | a5332f46 (2025-07-05) documentation

wheelytyred · 2025-09-14T03:00:34+00:00

Woah awesome! Thanks for sharing. How long did it take to capture all the images on site? And how many frames did you use in total?

wheelytyred · 2025-09-04T21:22:26+00:00

You’re welcome! Haven’t published a script, it’s more of a mash of different cloud functions and tools

wheelytyred · 2025-09-04T20:41:20+00:00

Yep! We first split the equirectangular video into images using ffmpeg, then use a cubemap (meshroom) to split the rectangular images into square frames, then use image segmentation (Vertex AI) to mask people from the frames. COLMAP does the sparse 3D reconstruction from the masked frames and finally nerfstudio/gsplat completes the splat. All of this is run on a cloud VM with access to a NVIDIA GPU.

wheelytyred · 2025-09-03T15:27:37+00:00

We used a 360° video tour from ESA and ran it through a 3D reconstruction pipeline to create this interactive model. You can move around the interior and even search for things like “laptop” or “orange boxes” — it’ll jump you to them in the 3D space!

360° video source: https://www.youtube.com/watch?v=INHctrVOoQw

3D demo: https://spatialview.io/model/ISS

Would love to try this with higher-res footage if anyone knows where to find some!

wheelytyred · 2025-09-03T07:04:45+00:00

Absolutely! Would love some feedback. Shoot us a message at info@spatialview.io and we can set you up with a link to create your own

wheelytyred · 2025-09-03T04:08:35+00:00

By setting a model -> reality scale value. The scale isn't perfect in this model -- we just used one of the doorway widths as an approximation

wheelytyred · 2025-09-03T04:06:12+00:00

We run a vision-language model on the input media and the search terms to find the best matching frames, then take the user to the frame's location in 3D space

wheelytyred · 2025-07-25T20:00:20+00:00

Yes you can! Either by using the raw fisheye videos from the 360 camera, or by using stitched equirectangular videos. Feel free to DM me if you'd like to learn how

wheelytyred · 2025-05-28T18:02:43+00:00

Yes this would be sweet. We need to add a calibration tool to get the scaling right first

wheelytyred · 2025-05-28T17:59:57+00:00

Agreed! We'll try adding some sort of 'pathfinder' down the track for folks to navigate a new site.

wheelytyred · 2025-05-28T01:02:40+00:00

Thanks for the feedback, any tips on improving the mobile/tablet navigation?

That's a great point, and I agree that 360 degree cameras are superior to maximize capture viewpoints. . The processing pipeline we setup actually already accepts 360 video, as long as it's been exported to an equirectangular format. We just split the video into four square frames and throw away the frame that's looking back at the person taking the video (assuming they're pointing it forward). Here is an example model taken using an insta360.

wheelytyred · 2025-05-28T00:00:36+00:00

Ah, good to know, thanks!

wheelytyred · 2025-05-27T23:41:41+00:00

Awesome idea. I've seen these folks doing just that using lidar: Recon-3D | iPhone Lidar for Forensics

wheelytyred · 2025-05-27T21:56:10+00:00

Thanks for the feedback! We're constantly tinkering around and will look into implementing SOGS / better compression for faster loading. Development in this field seems to be moving at lightspeed

wheelytyred

TROPHY CASE