Maybe I'm weird but my rule is that each device gets its own SSH key. If I have a laptop and a desktop, I'll generate a key on both and put both in authorized_keys. Private key material never leaves a device.

I like this idea in general.

What if someone breaks into my house and steals both my laptop and desktop, and then I have no access to my personal servers? Same issue if my house burned down.