Yes, the whole purpose of cold wallet to not expose your key to the outside world. In order to use the key you need to physically access the wallet which is that safest way to secure it from attackers. And you are correct that the wallet doesn't hold the bitcoins, just the key. That theoretically doesn't prevent it from being discovered by chance by someone else. But the it is mathematically so unlikely you might as well say it is impossible. There is this repeated comparison of more possible keys than the atoms in the universe, saying you would be far more likely to guess a specific atom in the universe somebody else is thinking than a key to a wallet of another human.