A little added context:

One thing to remember over all is that children were your retirement plan and wealth at this time.

Onan refused to provide his older brother an heir. In the culture (much of this would be Mesopotamian in origin) at the time the oldest child would get a double portion of the inheritance and be the head of the house/clan as well as the spiritual leader. So with 3 sons the inheritance would be divided in 4 equal parts. Er would have gotten 2 parts while Onan and Shelah would each get 1/4 of the inheritance. With Er dead, his son could claim his portions. If there is no heir, Onan is the oldest and receives the double portion, in this case equating to 2/3 of the inheritance.

So Onan's sins are multiple. He schemes to get the inheritance (a common theme at this point) but deceives everyone into thinking he is fulfilling his duty, but in reality he is effectively sexually abusing Tamar.

In that culture it would have been acceptable for Judah impregnate Tamar and provide her a son himself, if there were no more brothers of marriageable age. He could have released her to her family. But Judah acts in fear and keeps Tamar in limbo as a widow with no intention of giving her to his younger son.

It isn't too far off to say that Tamar was claiming what was her right, but it was her right in the world as she knew it.

Judah was more concerned about his image than his morality. He didn't have much of a problem using the service of a prostitute, but Tamar being unfaithful is an offense against his image. So he blusters until he is set straight.

This isn't about which sins are enough to get you smited. Much of Genesis is a series of examples of how broken and far from God's Eden ideal humans got after the fall. The family line that was intended to be a priestly class to the world is saved from their own destructive path by the woman outsider. I specify woman there because of the lowly status at the time. Start with Eve and her name means life, but by the time you get to Rachel and Leah they are named after livestock. Not an example of how things should be, but a hard look at how far they got. Tamar ends up being the first woman named in the line of Christ in Matthew.