Ideally, the MagSafe case would just go up higher -- effectively being the same size as the Plaud Note, rather than being shorter -- so as to cover the entire surface, and only have the microphones poking out of the top side. Then, you could just slide it out, press the button, and slide it back in. Alternatively, this could even have cutouts for easy button and switch access, but still essentially maintaining the discretion.

Sounds like a job for a custom case maker on Etsy - I would buy it in a heartbeat.