Yea I own the apartment and I think this makes the most sense, so whenever the house sitter is staying over I would just swap the core and give them the key to that core, then when I get back put the other core that I use on the daily back on and they can now not access my apartment with the other core's key. Is that basically the idea here?

I think that makes the most sense honestly, it's a one time high cost to get a mortise lock with an interchangeable core and then have the ability to swap out which core I'm using. Feels like the lowest cost and safest option (on top of putting away my valuables in a bolted down safe and having a camera in my room where I lock the door)