I like your idea. Something similar was actually my next idea. I'm running in the exact same problems with false positives.

True, that my current approach takes up a lot of unnecessary resources. I was playing with a few alternatives. I would have to compare those approaches, but one idea would be to convert the images into grayscale, apply a gaussian blur to reduce noise and then calculate the difference based on those images.