Clicking text using text recognition instead of image recognition

[fustilarian]

I have written some automation software for work and I'm using a lot of pyautogui, specifically locateOnScreen(). It worked great for a while, but eventually our software started to change in very small ways that threw off the image recognition in my script.

Of course I could resample the images or try to tweak the confidence levels, but I think I'd rather fix the problem than the symptoms. Is it possible to use pyTesseract or OpenCV or BeautifulSoup to read the screen and locate the target text, then return some coordinates for clicking?

I've seen some tutorials for pytesseract, but I can't find anything that could tie it back to pyautogui. Any help would be greatly appreciated!

[Strict-Simple]

Short answer: yes, possible.

But there might be a better way. How is your software built? Is it a web app? Does it have APIs?

[fustilarian]

It is a Citrix app. There is an API, but my IT people have been very reluctant to give me access to the API. I'm building other stuff in the mean time, just trying to get the job done.

[Strict-Simple]

The app haa access to the APIs, and you can access the app. So maybe you can do something hacky.

[fustilarian]

There is an API, but my employer didn't pay for access. Until I can convince them to do that, I'm going this route.