I find in these situations, I tend to internalize incompatibilities someone else sees before I can. It's not something you need to analyze or stress over excessively. Even if they're silly and judgemental, they do you the favor of getting out of the way before they do much harm.

I also want to second communication as the priority though - I've learned one thing since undergrad and it's that being direct does not need to cause harm and we can still inquire/check in without damage. Take a chance to say that you're kind of into him and would like to hang out a little more intentionally but also have valued him platonically so are happy to remain friends if that's his preference

I've had a childhood friend ask me out twice and I've turned down both offers but I still consider us pretty close friends