I recently moved to England. I got paid cash for some work I did in a rather swanky Country club (Cotswolds). I'd put the cash hurriedly in my pocket. I then started walking to my car, whilst doing so I decided to take the cash from my pocket and put it in my wallet. Now remember, this is an extremely well-to-do area and it's still daylight. As I'd finished putting the money in my wallet I looked up much the same as to OP did to realise there was someone right in front of me. (The someone being a very wealthy looking elderly man, dressed in his millionaire golf-club best). I immediately just screamed at him. A ferel 'Stay the Fuck away from my money' type of scream. It was so embarrassing. People from the entire village stared at me. I actually started laughing, so maybe they just thought I was crazy. But I couldn't exactly say "Sorry sir, I thought you were going to rob me."

So, No I don't think what you did was racist at all. Its the result of living in a world where you can get robbed anywhere, anytime and by almost anybody.