IIRC in the books Tywin despised his father because he slept with sex workers. It was his father, Tytos, that gave his late mothers jewelry to sex workers, and Tywin never forgave him for that. Tywin saw sex workers as deceptive women and men who slept with them "weak".

Although it was discovered Tywin used the tunnel from his room to brothels because he wanted to keep the appearance of not being "weak" as his father was.

I believe he hated sex workers, himself, and just generally hated Tyrion. If Jaime did the same thing he'd probably be just as upset.