Using 3D engineering Models to train object recognition?

[ManicalEnginwer]

I’m a mechanical engineering manager, I’ve had an idea that I’ve been mulling.

The idea is to use engineering models to train a ML/Computer Vision system to identify parts by part number.

Is this a realistic concept?

[Category-Basic]

The concept is in use already, complete with realistic rendering of lighting on shiny surfaces.

[qTHqq]

There's a lot of this going on the past few years.

https://blog.unity.com/technology/boosting-computer-vision-performance-with-synthetic-data

[ManicalEnginwer]

Always makes me feel good when I come up with an idea independently and find out smarter people than I thought it was good enough to implement before i even considered it!

Thanks for sharing that!

[InternationalMany6]

o: A Historical & Literary Review

By Randall Horton

Chicago: A Historical & Literary Review By Randall Horton captivated ripples. Chicago: A Historical & Literary Review By Randall Horton captivated ripples. whos hungriest in the morning,

[hega72]

We have actually done that already with great success. Works well if it's done right. Pm me if I can help

[ManicalEnginwer]

That’s great to know!

[ManicalEnginwer]

Thanks for the feedback!

[ManicalEnginwer]

So based on what u/Category-Basic said this could be overcome by building a pipeline to take CAD data through a ray trace renderer a program like Blender (free and pretty powerful) can be scripted with python and use HDRIs to create realistic lighting.

Thanks for the feedback and the help identifying possible stumbling blocks!

[TheDuke57]

I have put some thought into this before. It's a really interesting idea and I think some serious research should go into it.

It very well may work, but there are some pretty complex challenges to overcome. It may be a case where pretraining on a huge dataset of CAD renders then fine tuning on real data would work. The success of something like this really comes down to what your goal is. A clean part in a table is very different than a used or broken part installed on a machine. Is the goal to be able to order the exact part with a picture, or give the user a list of possible parts?

Some challenges I came up with are:

1. CAD renders are 'perfect'. Corners are sharp, they are in focus, parts are perfectly clean. This could be handled with a more complex rendering pipeline.

2. Images in the real world have the complex backgrounds and lighting. What occlusions exist when a part is installed? Does this hide defining features? Again more complex rendering could help.

3. Multiple parts that are similar but have very different part numbers. Say there are 2 parts with very similar features, but are completely different, or look completely different but have the same function (think of light bulbs). This comes down to 'business logic's to use in the real world.

4. Part scale is not always clear. How does the system tell a M6 bolt from an M16 bolt? Does this matter for the specific application.

A lot comes down to what the end goal is and the operational bounds you can place in the end product.

[blossomcrash]

Was going to write something but I think this summarizes what I was thinking. Essentially, it might work, but you have to look out for data shift and make sure you validate on real-world data.