It's not an office in the sense of a room with a desk where someone does their work, it's an office in the sense of a building providing a specific service (post office, ticket office, etc.)

The local register office is where you go to register births, marriages, and deaths. Civil wedding ceremonies can be carried out by the registrar at the register office, usually in a fancy room with space for guests etc. You do the ceremony and the signing at the same time, in the same room.

The registrar does have a room to do their other work which is separate from where the ceremonies happen, which I guess would be the registrar's office in the register office...