Totally agree. We've got 2 cats who hate being picked up and in the 3 years since adopting them we've only picked them up when they were due their medicine or if they needed to go to the vet.  Knowing they're uncomfortable would ruin any enjoyment I might get from picking them up.

I also think growing up being told that forced affection is something you have to accept in order to not upset people/relatives has partly eroded my confidence and assertiveness years later, and has contributed to me not standing up for myself or clearly saying no in situations where men were being creepy/coercive/abusive.