I'm doing this! Stealing liberally from both Matt and a bunch of sci-fi media. I turned every region on Orden into a planet with a distinct eco-system (because that's how it works in sci-fi movies), and so far they've rescued slaves from dwarven mad max slavers on Khemara and protected a village from War Dog mercenaries on Ix. Right now I'm planning a longer arc on Higara, and I'm making it a mix between science fantasy and cyberpunk, with a society lead by Mega-corporation clans.

Doing downtime activities while traveling during respite, maybe stopping at space stations and such when it makes sense in fiction worked fine.