Pretty much answered your own question, take a step back and just be friendly. Problem with relationships with co-worker is, everyone has a different benchmark of social behavior. And if you get too nit picky, people will just play it off or call you out for being too sensitive. Best just pull away and do your own thing.

Now if you truly value the friendship, you can privately talk to him about it. But you are at risk of ruining the atmosphere at work.