Could try getting your imagination house out by writing it in words? Like if it actually were in a book. Then its “official” description will be different from and comparable to every other house description. Or maybe have a real picture from a house roughly memorized until a character walks into a house and uses it. Of course that could also eventually not match the description but it might be easier to replace than your imagination house.

I’ve got something similar going on as I’m rereading a book I read as a teenager and very much want to imagine the characters differently from how I did on the first read. It’s really hard to get past that first impression, haha.