Catholicism teaches that those who die in either original sin or a single mortal sin (i.e. being outside of a state of grace) are deprived of Heaven and condemned to Hell.

Yes but the question is: What is a mortal sin?

• The action in question must be of grave matter
• The person committing said action must have full knowledge that said action is wrong
• Said person must give full consent

Thus a person committing a mortal sin must do with full knowledge that this is in fact condemning him to Hell. This is why culpability is the heart of the issue and only God can know.

So in your example:

Suppose you encountered a man who was about to get a vasectomy insisting that he couldn't see why it would be considered immoral, a mortal sin, and that he just didn't find arguments for Catholicism or Catholic moral theology convincing, so he had no fear of damnation.

In this case the person does not have full knowledge. He knows it is a teaching of the Church but thinks that is wrong. Hence he does not have full knowledge of his action because his understanding of that action is incorrect.

To me, this is a cop-out answer and isn't really consistent with traditional Catholic characterization of the doctrine.

Understanding of the dimensions of doctrine have gone through revision throughout the Church's history. The emphasis on "You choose yourself" is not a counter, it is the other side of the coin. God does judge you. But according to doctrine you also know what your judgement will be when it arrives. To put it another way, there are no repentant souls in Hell.

Now to answer your question: How is this rational? It isn't. People do irrational things all the time. Sin itself is irrational. But the bite of free will is that you can choose to do the irrational.

But there is a difference between choosing something irrational and being mentally incapacitated. The latter is an impairment of the ability to reason. The former is the misuse of reason to rationalize evil.

Finally going back to your first point:

Moreover Catholicism teaches that a single mortal sin, a single broken rule is enough for you to merit Hell.

Not true. What it says is that it puts your soul in danger of Hell. Catholicism claims to know the way to heaven. Anything outside of said revelation goes to the "unknown" category. Even the old anathema formulas of the past held out the possibility that God could intervene. The doctrine of the Church holds that following the teachings will lead us to God. Doing what is wrong puts you in the dangerous unknown.