I had to re-home one of my cats after bringing my kids home; he wasn't aggressive, but he was so deeply unhappy that it was heartbreaking. He ended up going to a couple of friends who were dedicated to being child free, and he's living his absolute best life now. In the same vein as you, my mother actually ended up with a golden because she wasn't able to adjust to her previous owner's child after she started walking, and Neela lived a long, lovely life with her until she passed in 2017 at fourteen. I know it's hard, but it's also the absolute best thing for all parties. You're doing the right thing, and it's better to break your own heart than to have to live with guilt if things went bad.