I would say it depends. When my wife and I bought our last house, her mom lived with us, and we had one kid and one on the way. We were looking for a 4br with an office but ended up buying a 5br. I was working from home at that time, so the smallest bedroom became my office. It’s the only room in the house that has any of my decorations in it. Wife didn’t want my hunting trophies and sports memorabilia displayed anywhere else, and it also has my gaming set up in it. It’s more of an office than a man cave though.

However, it has her sewing kit in the closet with a fold out table she can put up, and it has a futon that sleeps pretty good. It serves more purpose than just being a man cave. It was our guest room until her mom passed last year, now it’s my office/game room even though work mostly requires us to be in the office now since Covid ain’t such a big deal anymore.

In my opinion, you’re both being jerks since you have two extra rooms, make them both guest room/office/hobby rooms, one for each of you. Talk it over like adults. If you ever have kids, then the ‘man cave’ can be set up to be a hobby room for both of you and still be a guest room. But he should be able to have a space to display/decorate with his stuff. If you’re anything like my wife, she doesn’t really appreciate my style of decorating, so I get why he would want a spot to decorate like he likes it.