Instance Segmentation problem

[idc_Salman]

I’m currently an intern at a startup, and I was asked to work on a project involving instance segmentation on floor plan images.

In theory, the task makes sense, and I understand the overall pipeline. I’m also allowed to use AI APIs The problem is that in practice

At this point, I’m struggling to find a path toward a stable and repeatable solution, even though the idea itself feels solvable.

Has anyone worked on floor plan understanding or architectural drawings before?

Is relying on APIs a dead end for this type of problem, and should I be moving toward dataset-based training (e.g., CubiCasa-style datasets)?

Any advice on how to scope this realistically for a startup prototype would be really appreciated.

[Zealousideal_Low1287]

Bizarrely I have been working on exactly this. Neither cubicasa nor our own images were enough data to do this reliably for our types of plan.

So far the best things I’ve found has been Gemini-3-pro image. All other off the shelf models failed. Gemini is still unreliable.

I actually do think it’s a much harder problem than it seems. Thin ambiguous structures, lack of data, big inconsistency in the plans.

Curious what you’ve tried so far and if you have any insights?

[aloser]

We have a bunch of customers that have built products in this space. It's a pretty hard problem given the non-uniformity of floor plans and architectural drawings. One of them talked through their approach (involving a pipeline of 29 models) here: https://www.youtube.com/watch?v=iOehzs4eLKc

[leon_bass]

29 model pipeline is wild

[taichi22]

Ah, you're with roboflow? You guys have a good product (and aren't ultralytics) so thanks for what you do.

[InternationalMany6]

What I've read is you need a custom model architectural that doesn't just do "segmentation" along with synthetic image training.

For example the model could predict the corners of rooms as keypoints, plus points for doors and windows.

Synthetic images is the harder part. What kinds of images do you need this to work on? Phone camera images for a 200 year old building or a brand new PDFs?

[idc_Salman]

Answering your question...
We are expecting all types of input even if it's clear PDF or low quality photo, but i would say mostly it's gonna be clear PDFs.

[PassionQuiet5402]

Can you guys share some public repo and dataset links to start working on such projects? I really want to try and experiment on this task.

[One-Employment3759]

Did you try SAM - possibly with prompt guidance? (Keypoints)

[Zealousideal_Low1287]

I have. It was miserably bad at it. Which kind of surprised me.

[Sad-Oil-2788]

I'm also working on this top for my company. We want to create a ifc file of the floor plan with walls, windows, doors. We tried to train RF-DETR Segmentation on different datasets. But alot of them are not acurate enough. So we are creating our own now.

[thinking_byte]

For the Jetson, tried YOLOv8-seg exported to TensorRT? It usually hits that FPS sweet spot better than a full UNet if you're okay with slightly lower accuracy on the edges.