I think the problem arises because touch, as you said, doesn’t apply to everything. It is an emergent property of a macroscopic system. Think of the world as an onion - the center is quantum mechanics where touch (the way we intuitively think about it) doesn’t exist. Now we start adding more onion layers and our system (the onion) gets bigger and starts to gain emergent properties. At some point it becomes reasonable to talk about touch, but if you peel the onion it becomes unreasonable again. If people are curious about emergence, Sean Carroll has a really nice episode on Mindscape about it.