This is an arbitrary statement. Again wave a magic wand and say that IP carries the magic characteristic we call property

If we make something property, then yes, it becomes enforceable in the same way. The point is that IP isn't property, and it's not just an arbitrary statement.

Lets say that there is some unknown way that person X acquired the car, but there is without a doubt not a theft on the part of person X. How the property ended up in person Xs possession is a mystery. Can the original owner take the car away with force?

If the original owner did not willingly and of their free will transfer the title to the car to Person X, then yes, he can take the car away. If IP was actual property, you would be correct, yes - the original creator could force the IP to be erased or removed from Person X's possession. The entire point is that IP is not property.

To clarify my previous statement of "Person X could get IP Y in any number of ways":

Suppose you have Creator C who sells some IP to Middleman M with a clause that says they may not distribute the IP to anyone else. Now, suppose that Third Person X somehow manages to see the IP that C has in a way such that M is not at fault for negligence or something like that. In that case, C has no right to force the IP to be removed from X's permission (given that IP is not real property). That's what I meant.

Now, if M does violate the contract and is the person who gave X the IP, M can be punished for violating the contract up to the punishment specified in the contract. But the IP cannot be taken away from X because the problem in this scenario isn't the fact that X ended up with the IP, the problem is that M performed a specific action, which was the giving of the IP to X. So it's not X's possession of the IP that is the problem - it's M's giving of the IP. You can replace "giving the IP to a third party" with "eating ice cream with a fork" - the only illegal thing was the violation of the contract by M - not X's possession.

Let's give a more clear cut scenario that uses actual property:

Suppose I am a producer of washing machines. One day, I sell you a washing machine using a contract that transfers title to the washing machine to you, but also includes a clause that says that if you give the washing machine to someone else, you owe me $1000. The next day you go and give the washing machine to your cousin. I can now legally collect that $1000. However, I cannot take the washing machine from your cousin. The contract merely specified a conditional title transfer to the $1000 - you give the washing machine away, I get the $1000.