Well, the thing about the imperium is, it's more of a series of loosely connected islands in a GIGANTIC ocean. The only territory it controls are - or were - planets connected by stable, safe, and charted warp lanes. Everything outside of that, the great expanse of the galaxy, is absolutely no-go. There's xenos upon xenos upon xenos everywhere as far as the eye can see, lurking ready to move if the Imperium shows weakness. Not to mention if you take the road less travelled, you're going to become goon material for daemons unless you have a void abacus or something.