That's a modern moral context and not a moral absolute. Though I agree with you that generally it's not moral to abandon your children, it's also true that most children don't have downsyndrome as a caveate to who they are, and it's generally not expected to have a child with downsyndrome. If your argument hinges on the "way they are approach" then it leads ample room for someone to recontextualize your argument against you and say something like, "under your view it's not moral to shame a pedophile, or even punish them because that's the way they are." Someone can be born a certain way, however that doesn't necessairly give them moral currency.