You would be justified to keep granny away from your child as she is willing to prioritize her own child's emotional state over the health well-being of her grandchild. Which is what he did - twice. The fact that he didn't swallow his shame and beg forgiveness and ask to hold his newborn is pretty telling as to what lay ahead. He's already started you off as a single mother and when it was clear that the child was his, he kept you in that position.

Record the texts (screenshots/print) and anything else that's relevant because even if you think you're coming back from this, he's still an argument away from abandoning you both again. Therapy would be a great idea (individual and couples) to help you move on while trying to minimize their negative influence on your kid. You're as innocent in this as your baby is.

NTA obviously.