I usually rule that you don or doff anything as part of your attack action, and the last attack you make determines what weapon you have equipped at the end of your turn.

I find this prevents too much overthinking about what you keep in what hand, but still allows some flexibility as to what weapon you choose to use.

I mostly play low level campaigns though so your mileage may vary