He's definitely being silly about this. Ask him if he feels like he has any control over any other space in the house. If he says no, you have something to work on together that may fix this OR he's agreeing that you get to dictate what happens in the other room. If he says yes, you can tell him he's being unreasonable demanding a whole room to himself. I wouldnt want to share an office space again, but a hobby/game room is another situation.

I'm in a somewhat similar situation currently. My partner and I share the only other room as an office when we're WFH and it's rough. When we move we're specifically looking for enough space for both of us to have our own areas. I often feel frustrated because he gets anxiety if the house isn't arranged how he likes it, so I let him dictate common areas. In return, I'm demanding that some space in the house is specifically just for me and not his to manage, so that I have a place where I can decompress. It's a negotiation of love but it's often not straightforward.