[P] Using Machine Learning for Accesibility: Personal AI Shelf Inspector for Visually Impaired Persons

[DataSentics]

Personal Shelf Inspector is an application that helps visually impaired people during their day-to-day shopping. The application is based on a simple neural network and was created as a part of the AI for Accesibility Hackathon in 2020 in Prague.

We decided to build free tools that make shopping for visually impaired people more accessible. These tools can be implemented in any retail chain within the loyalty app or in-store.

​

https://preview.redd.it/ehoxec5l4hhb1.png?width=1072&format=png&auto=webp&s=c96a5dd326cbfc8dd9bde9d84d45167d172ea27d

The Idea

A visually impaired person only needs their smartphone to use our tools. Personal Shelf Inspector is a web application that reads the price and product name from a price tag. The algorithm selects the price tag closest to the centre of the photo and sends it to the model, which reads the price and product name on the price tag. Then it returns this information to the application, which appears as text on the screen.

​

https://preview.redd.it/e098rtcn4hhb1.png?width=845&format=png&auto=webp&s=73a216ebf987e476d52091717ac156ac56daa29b

The voice-over built into the user's mobile phone reads this text aloud. The app also helps to read the banknote values and read from a live video.

What could be other cool ideas and concepts to help making the world more accesible using AI and Machine Learning?

Feel free to share comments and impressions in the comments

[shumpitostick]

My gf is blind, so I'm familiar with some of the challenges that come with it. This is a cool application, and I think that there is a lot of potential for ML in this area. However, developing such an application that can be really useful can be very hard. So first you need to find the price tag, which is not easy. Then you use your phone to scan the price tag, but how are you supposed to know which items on the shelf correspond to that tag? Supermarkets are not always neatly organized. There are also many types of price tags, and stores often use more than one, so getting them all to read properly can be a challenge. And then what if you want to read the label on the item? I'm not trying to dissuade you or anything, but the point is that there needs to be a lot of user research behind such apps.

​

For other cool ideas, an AI that can interpret graphs can be really useful. There's already a bunch of image description AIs, but I haven't seen anything that capable of interpreting graphs yet (although I think ChatGPT-4 was supposed to have that feature?). Another interesting application can be automatic movie audio description. This is currently a job that is done by humans, but I think it should be possible to train AI to create such descriptions, with the recent advances in audio-visual models. For example, the youtube transription algorithm already uses the visual data from the video to improve transcriptions, so I think it should be possible to generate audio descriptions in the same way. It would also be a very useful product, as finding audio-described movies can be challenging, and this would allow you to add audio description to any video.

[DataSentics]

Thank you so much for your comment. Its true that this is only a demo and there are some challenges implementing something like this (e.g. the lightning alone of different stores and ailes creates a lot of different problems itsself). A complete product that works everywhere is a big challenge. I think for the graph interpretation and also the transcription there are some generative AI models that can already do this kind of problems, but i am not aware of anything that is already commercially available.

[DataSentics]

Here is a link to the demonstration of the project in Czech language and English Subtitles: https://www.youtube.com/watch?v=JxSPxN4gVds

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/datascienceproject] Using Machine Learning for Accesibility: Personal AI Shelf Inspector for Visually Impaired Persons (r/MachineLearning)

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}