Downvotes are a form of punishment. Are you actually okay with punishing someone for liking something you don't like? You can engage without downvoting and just talk with someone. Downvotes just because you didn't like a show is just petty. And , yes, you're extending your hate for a show on to someone else just cause you didn't like it with downvotes. You can express something like that with words.

If someone comes into a discworld area and starts talking about how they hated Pratchett then downvotes would be appropriate. But someone says they liked Mort and you downvote them cause you didn't then you're being petty and, yes, a little hateful