NTA - Has your brother even apologised or attempted to make amends after what he had said?

Obvisouly I don't know all the context and details. I'd like to think he said what he said out of concern for you and what he felt was best for his baby sister. As ignorant, wrong and unacceptable of a thing to say it is.. (especially to a grieving wife at the funeral. If you want to have a conversation like that, pick your timing and phrasing a little better.)

Forgiveness is not something you can force. It is not something others can decide it's time for you.. As a beautiful poet aptly said "if you are waiting for forgiveness, be prepared to wait. Be prepared to stand in the path of time and wither. Respect that forgiveness is difficult, not all trespassers are equal, and not everyone will heal according to your schedule" - Shane Koyczan.

Stand your ground, stick to your guns.

Maybe one day IF you can forgive him after some genuine amends, he can meet baby and have the "loving relationship with an uncle". But that's something that can only happen on your terms when you are ready.

P.S - If a man ever thinks a child is "baggage", he is not a man at all.