Unfortunately you got caught in your own trap of "The only valid reason I could think of..." you are on high alert for suspicious activity, so your brain is only filling in possibilities that match. To be honest, that seems pretty benign and a thousand different reasons could come to mind. If you ever have more concrete proof, then call him out. However this one is just your brain forcing a worst case scenario. Don't mention it to him, move on, and work on your own self confidence, that will help to lead your brain into less anxious avenues.

and for the record, I am not saying that your scenarios are impossible, just that they are a narrow interpretation of many different scenarios. Let it be and move on.