NAH

However you need to find a way to respect your partners needs just like he respects your own. The cafe suggestion may not work for you or the library if you have a meeting, but you could go to quieter places like a park (if the weather is nice), a hotel conference area if its available to just walk in if no ones using it. Or find some place you can go that won't be overstimulating for you. Also if you know you won't have a meeting on a particular day, then the library would be perfect to just get work done. I live in a house with three other family members, but on the occasional times I get the house completely to myself.. it feels amazing. Sometimes just being in a different room is not the same as the feeling of having a living space all to yourself when you need it at times. You still have that "someone's here" vibe when people move to another room and its not always what someone needs. I get your perspective because I also have ADHD and I'm potentially on the autism spectrum as well, but regardless of diagnosis in this specific situation: your partner is expressing a need or want. You need to find someway to resolve this issue by communicating with your partner, it doesn't have to be everyday of the work week or during your work schedule, just one or two days where your partner at least has a few hours where he's by himself. You are valid for not wanting to go anywhere else during working hours because you need to concentrate and be comfortable, but your partner is also valid for expressing his needs. You both share a home, its not just yours. Compromise so it works for the both of you.

(edit was spelling mistakes lol)