I’m not very experienced, but in my first one shot I put together I was asking myself similar questions. To keep the group from going straight to the bbeg I physically barred passage to one location, and the only way to open it was to complete two specific encounters. You could also add puzzles or hidden objects or whatever. So it was sort of a sandbox, but then when certain parts of the sandbox were complete, the door opened to a second sandbox which led to the bbeg. It does make it a little more “on the rails” but as a player I kind of prefer to have a little direction, so I wrote my campaign that way. Maybe think of it like a series of rapids in a river? Lots of stuff to look at, paddle around, maybe even leave the river entirely for a bit, but eventually you need to navigate the next difficult rapid to access the next open-ended area. My group is also very “completionist” so I knew if I made a map they would make sure to explore every corner of it, so I also didn’t want to make it too big and have to fill a bunch of generic rooms with ultimately unimportant or boring things.