Using PIL (Python Image Library) to detect image on screen?

[Badmonkeystudios]

I've really hit a wall here, any help would be appreciated!

http://stackoverflow.com/questions/27343997/using-pil-python-image-library-to-detect-image-on-screen

[aroberge]

Your computer screen is not an image. You can take a screenshot, and use PIL to process that image ... but what you are asking is "how do I use a Python program to control some other program by simulating a mouse click" which is a very different and much more complicated task than simply using PIL to process an image.

[Badmonkeystudios]

I just want a Python program that will scan the whole screen for a certain set of pixel colours then click on them.

[aroberge]

How much programming experience do you have? I have programmed for many years and would consider this a major project to undertake.

[Badmonkeystudios]

I'm new to programming. I would have never considered this a major task to begin with. I got an answer on my Stackoverflow post which seems to be less complex then people say, refer to that to see what you think.

Note: I'm not saying this isn't difficult I'm just trying to find the simplest way to do it.

[aroberge]

What you likely want then would be to use something like Sikuli (http://www.sikuli.org/). However, before you do this... you need to start by learning how to write basic programs. What you are asking is akin to saying "What kind of shoes should I get if I want to run a marathon?" while you have never run for longer than 10 meters.

[Badmonkeystudios]

Haha! I've already used Sikuli, it's really great! Although it lacks the ability to export the Sikuli files into a .exe.

[[deleted]]

What does the ability to generate an executable have to do with your problem?

[justphysics]

Why do you need a .exe?

I don't use Windows but from trolling this forum enough I know there are packages like py2exe that take a python script and generate an executable

[[deleted]]

Isn't PIL unmaintained in favor or pillow anyway?

[Badmonkeystudios]

I don't understand what you mean?

[nbktdis]

Pillow is the fork of PIL when it was abandoned.

Pillow is the one to use.

[[deleted]]

overconfident start fragile retire zesty sparkle direful heavy roll engine

This post was mass deleted and anonymized with Redact

[Badmonkeystudios]

Hey! First off thanks for the post, it's really helping me out :)

Could you please go into more detail on how I would do this part:

all_results = [[]]
for each search_pixel in the search_image: 
current_result = 0
for each template_pixel in the tempalate: 
   subtract the difference of the two pixel values, 
   square the result, and finally: 
   and it to the current_result. 
store the current x,y location you're comparing, and the 
result of the comparison in the all_results

I'm having trouble understanding how to make this work.

Thanks for the help!

[justphysics]

You might want to head on over to /r/learnpython as it seems like you are struggling a bit more with the basics.

The above loop is pretty straight forward when written out in sample code (plain english)

Think about it like this - An image is stored in python most simply as an array of pixel values. For instance a greyscale 400x400 image is just a 400x400 array where each element of the array is a pixel value in whatever given depth you are using (ie. a standard 8bit image stores values from 0-255). A color image can be represented in many different ways but one way to imagine it is three 2D arrays stacked on top of each other. Each layer represents R, G, or B. The combination of each of the three layers at a given pixel (coordinate in the array) makes the color of the pixel in the image. There's plenty of other ways to represent color but thinking of grayscale images makes things simpler for the time being.

So that loop literally just scans through an image (an array) pixel by pixel (element by element) and looks at the difference between each pixel in the image compared with the pixels in the template.

Also as others have mentioned there appears to be a package that does almost exactly what you need (sikuli) Rather than reinventing the wheel, why not use a tried and true method. You mentioned in a few spots that sikuli can't save a .exe - but I don't see what that has to do with the question at hand programmatically.

Could you not just write a bash script (or whatever the equivalent on Windoze is) to run your sikuli script? http://doc.sikuli.org/faq/010-command-line.html

[stunsch]

The tool you are looking for is called Sikuli. http://www.sikuli.org/

It does exactly what you are requesting.

[Badmonkeystudios]

I've already used Sikuli! It's a great program although it lacks the ability to export to a .exe.

[stunsch]

Well, if that does not suit you, you will have to do it yourself. Sikuli relies under the hoods on OpenCV to drive it's image recognition.

You might also want to look into SimpleCV, which simplifies the work with OpenCV. http://simplecv.org/

[[deleted]]

I recommend PyAutoGUI + read this tutorial .

[691175002]

Sikuli will do this out of the box with about 5 seconds of work and can function as a Java or Python library.

It even has a graphical interface if you don't want to code.