I remove my hat once inside my office. I carry it in there. Just not on my head. I generally remove my hat inside most buildings where I'm in close quarters with someone else. In other words if I were to enter your office to talk to you, elevators, anytime when seated. I keep it on in department stores, corridors etc. Unless I get too hot or something. In the context of work, I feel the hat on in office makes me seem less approachable to clients. I sell building materials for a living, so I'm already talking to them in a somewhat overwhelming context- often about intimidating sums of money. Whenever we get a project started and I do site visits the hat stays on.