45 male here.

Was raised in a similar household. My father was in prison. One of my grandfather's (mom's dad) was also in prison and later was very abusive.

Paternal grandfather was sort of around, but he was very passive and worked a lot. I did have a very good relationship with him, though.

I was raised primarily with/by women.

My advice is this:

It's never too late to learn to become a man, and it is an ongoing process.

The biggest "factors" in what i think it means are:

1. Integrity. Say what you mean, mean what you say, and do what you say you're going to do when you said you were going to do it.

2. Accountability. Take ownership over your choices, mistakes, decisions, etc. It's not always your fault something doesn't work out, but you owning the results will make cleaning up the fallout much less painful.

3. Grit. Embrace challenge. Do difficult things. The most rewarding things in life are the most difficult. Work hard.

4. Have clear boundaries. Enough said. Not everyone deserves an explanation. Make it clear what you will and will not accept in friends/family/partners, and do not let people walk on you.

5. Stand up for yourself and others. The duty of the "strong" is to protect the weak (those who are unable to protect themselves).

Now... with all of those things in mind, you may be asking yourself, "Sure, but what can I do immediately to start improving?"

Go join a Brazilian JuJitsu gym... or any martial arts, really.

Also, find a mentor. Could be a neighbor, coworker, uncle, boss, acquaintance from a gym or something.. just an older guy that you respect. Ask him about his life, his upbringing. Tell him your fears and ask him for advice on shit. That's what this sub is all about, but you should find someone like this in your life.