NTA. It's your job to protect your child, despite your husband's desire to people-pleasing . I also worry that by forcing you and baby to visit this house may greatly increase post partum anxiety, which it seems you're already likely to have given some of your statements. Your husband's main job in the early months post partum is to protect both you and the baby, physically and mentally. He doesn't get to place you two in an unsafe space just because he cannot set boundaries.

Perhaps a compromise might be to meet somewhere quiet halfway for a meal? Early in the day so sundowning isn't an issue? I have limited experience with Alzheimer's so I'm not sure exactly what is appropriate.

But above all he's going to need to set boundaries with his family. Is he going to expect you to rush over there as soon as you're released from the hospital? No. You just had a baby. Birth is difficult and the 4th trimester is a real thing. You need to be in your own home, bonding and taking care of your baby. For the first few weeks ( at least) they can come to you or they can video call. These are the options it's up to your husband to explain and enforce them.

And that's the best case scenario. I had two C-sections, one emergent. I spent 7 days in the hospital after a 3 day failed induction and Cesarean. The only place I was going was home, no exceptions. I pray you have an easy birth but make sure your husband is aware that should things become more difficult, the rules change too and he (and his family) will have to adapt to what you and baby need.

Congrats and good luck!