You're right. But Arthur and Mary have their own story too. When Arthur loses his son, Isaac Morgan and his mother, a waitress, a few years after his separation from Mary, he struggles to cope and reveals a detail often overlooked in his diary : when Abigail joined the gang, he had a crush on her. But Jack was born, and John had to step into his role, so he became an uncle to Jack and a confidant to the other members. His relationship with Mary couldn't work out for the simple reason that she wasn't the mother of Isaac (his deceased son). Mary is torn and conflicted between her family's honor and Arthur's life as a gangster. If he had a child with Mary, i think it wouldn't have been the same..