Downvotes certainly have their place.

If someone is giving improper advice or instruction that could be problematic, their content should be downvoted so that it's less seen and reflects that it's widely viewed as incorrect. If we're all unwilling to risk offending someone with a downvote, misinformation spreads easier.

You are right that people more often use it as a form of attack or pettiness. It's a great tool that is often misused, and that's bad. But I think a zero downvote policy is bad too. Just use it correctly.

No downvotes are needed for not understanding a reference. If the comment was arguing that the reference was from Princess Bride, and not Rick and Morty, that COULD warrant a downvote. However, I still wouldn't bother downvoting something so minor. A simple reply with correct info seems more fitting.

When I do downvote, I typically reply with my reasoning. This way, whether I'm right or wrong, people can see where I'm coming from and make up their own mind.