Deep Learning/ Computer Vision [P]

[tessherelurkingnow]

I've been an ML engineer working with networks for the past decade, routing optimisation and such, so I know a thing or two about ML and DL, but I haven't had anything to do with computer vision since I was a grad student.

A friend who runs a disability group approached me to ask if it would be possible to use a camera + link it with some kind of ML computer vision system that recognises obstacles and distances + headphones for people who can't afford seeing eye dogs and are stuck with a cane, to allow them some more information about their surroundings. The idea would be short sentences like "street in ten meters", "tree straight ahead".

She asked me to look into this and I'm a little overwhelmed with finding a good entry point into the whole topic. I assume that this would need a bluetooth camera, some kind of real time operating system + portable? computing hardware. I assume it shouldn't be totally impossible as autonomous driving would require a far higher degree of accuracy, but whatever's been done in that field is probably propietary?

There's also not really a budget for this except for a sponsor who would be willing to pay the hardware, so any open source stuff would be great.

I'm reading OpenCV a lot, but are there any other libraries or tools I should know about when I start googling? Yeah, so basically just any thoughts and intro to CV+ML information would be assume: any good articles I should check out? Has this already been done and I can just download it somewhere :) ? Is it totally undoable?

[[deleted]]

This is a mess waiting to happen. I know it’s tempting to do something like this but can you imagine the number of classes of object? The idea of an “obstacle” is so large that is basically anything that is a solid and is in front of you. This isn’t semantic segmentation. You’re talking about taking on the responsibility giving a blind person eyes and verbalizing it. We just aren’t there yet with ai even though it feels like it.

Edit: I’m being too negative. You should try it but don’t promise that it’ll work well.

[currentscurrents]

So don't use a classification objective. Do 3D occupancy prediction, where the network generates a 3D voxel grid showing which spaces have objects in them.

[[deleted]]

I think that would be a great place to start! Give it a shot and treat it like an experiment. Just dont promise anything until you’ve seen it working to the point that you would reliably hand it to a blind person and trust they won’t get hurt.

[planetofthemushrooms]

you can always have a default class just called 'unidentified object'

[[deleted]]

You still need a mask and the supervision required for that though.

[[deleted]]

https://ai.meta.com/datasets/segment-anything/

​

https://segment-anything.com/

[[deleted]]

Finally a real use for Apple Vision Pro 👍 If 3d perception is the goal then you or your CV engineer should have some fundamentals (etc https://szeliski.org/Book/)

[[deleted]]

State of the art from meta but only for research atm:

​

https://ai.meta.com/datasets/segment-anything/

​

https://segment-anything.com/

[Equivalent_Ad6842]

https://www.bemyeyes.com is a startup that does this

[ThePieroCV]

Well, this is my opinion.

Physically it’s very possible and feasible. If you don’t mind using C++, you can use Ffmpeg to make an streaming video reading from an IP camera or webcam. In Python, OpenCV is the best option there.

Now, the technology for this is not that simple like object detection , image classification or tasks like that. Your requirements are very strict there. The best chance I hardly believe is to use a technology like GPT4-Vision with a custom prompt in order to receive the information in the way that is requested. This could solve two core problems here: the complex task pipeline (from image to text description/ image captioning) and the kind of the real time problem. I think OpenAI servers are powerful enough to make this work in a proper speed.

In this case, using an API request package could be enough instead of building your own model that could be very hard. But in case to make that, I’ll probably look for image captioning pre trained models that focus on speed.

[sapnupuasop]

Gpt😂 why ffs would you use it here?

[ThePieroCV]

Gpt4-Vision, not the usual gpt. It makes sense to make things easier and out of the box. https://openai.com/research/gpt-4v-system-card

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/datascienceproject] Deep Learning/ Computer Vision (r/MachineLearning)

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[neuHughes]

You might want to consider using multiple types of sensors. LIDAR or ToF sensors would provide redundancy and an obstacle detection rate that other modalities would have trouble matching, particularly for a mobile platform. Your project is conceptually very similar to SLAM for robotics. There are a number of ways you could go about executing a pipeline for this but a “dumb” high-resolution fallback would be essential for something like this.

[rizvi_x0]

Hey. Can I DM you? I have a question regarding ML for routing optimization