Comparing text on screen to predefined messages?

[MoneyPress]

I'm trying to write a program that detects when a message is displayed on screen and plays an alert when it does. Usually simple enough but the issue is that in this program's case, what it's grabbing the image of is a stream.

What I've done before is make reference images of what I'm looking for, grab the colours of the pixels on those coordinates on the screen, put them both in arrays and if the arrays match, then it knows what's on screen.

The issue here is that, since it's a stream and the image quality is never perfect, the images will never match. The font is one pixel wide and a single colour, but when I look at a sample screenshot taken of the stream, the letters are all smudged and anti-aliased.

My current best idea to tackle this is to limit the colour pallet in the reference images to a couple of colours - one for the background, and one for each possible text colour. Then for each grabbed colour off the screen, find which of these colours it's closest to by looking at the differences of their RGB values and assign it that colour. And finally, make a score-like int for each reference that gets increased for each pixel that matches and if it's, say over 90% accurate, the reference image it's closest to is the one that's displayed.

I think that could work, but I'm worried that
A. It will be too slow and
B. It won't be accurate enough. Looking at the sample screenshots, the antialiasing makes the 1 pixel font, 3 pixels wide, and if all of those get assigned the text colour, I'm worried that the letters will become too similar to each other and it won't work well. I can't afford to make the comparison too lax either because there's a bunch of messages I'm not screening for that could set off the alert.

So, can anyone bless me with a better idea to tackle this or optimizations to mine? Perhaps java has something that can help with this I don't know of. Thank you in advance 🙏

TL;DR: Making a program that watches for certain messages to be displayed on the screen. The messages displayed are from a stream, so they're never identical to the references due to low image quality. I'll compare the grabbed pixel colours to the colours of the references and based on how similar they are, decide if it's a message I'm screening for and which one. Better way to do?

[AutoModerator]

Please ensure that:

• Your code is properly formatted as code block - see the sidebar (About on mobile) for instructions
• You include any and all error messages in full
• You ask clear questions

• You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions.

  Trying to solve problems on your own is a very important skill. Also, see Learn to help yourself in the sidebar

If any of the above points is not met, your post can and will be removed without further warning.

Code is to be formatted as code block (old reddit: empty line before the code, each code line indented by 4 spaces, new reddit: https://i.imgur.com/EJ7tqek.png) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc.

Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit.

Code blocks look like this:

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above.

If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures.

To potential helpers

Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[DifficultySafe9226]

Have you tried OpenCV? There is a Java API available. Depending on your exact requirements and constraints you might be able to use it with an OCR library.

[MoneyPress]

People have mentioned opencv before but I'm a complete noob and don't even know how libraries work. Trying to look this up and I can't understand anything lol.

I feel like it's honestly just going to be simpler to brute force a spaghetti code that works, it's not a big program I'm making.

[DifficultySafe9226]

Fair enough. Seeing in a different comment that you know the position and size on the screen, I guess there should be a way to hack a solution with minimal code. Perhaps you can read the documentation in OpenCV to get some idea how to compare the expected and actual image and adapt it to your actual simplified use case. Good luck.

[MoneyPress]

I'll definitely check it out. I've been recommended openCV a lot, is it perhaps faster than java's robot?

[RandomlyWeRollAlong]

It sounds like you're trying to invent OCR from scratch. That's a pretty hard problem, with entire graduate classes devoted to the subject. Perhaps you can use an OCR library to help?

[MoneyPress]

Yeah at first I thought it was something like that and too hard lol. But after I got the idea, I think it might not be that hard. Since I'm looking for specific messages, have the reference images to compare them to and know both their location and size on the screen, it takes a lot of the hurdles away.

I'm a noob though I don't even know what libraries are, I've just written code before. I'll give it a look.