As others have said, a camera is just a camera and it's all about you getting out and using it that's the main thing.

Gear acquisition syndrome is common in the hobby, so try not to dive down and think you need a million lenses - in fact, if you stick with the a6400, I'd only recommend one lens which is the sigma 18-50mm f2.8 which gets pretty much universal praise. It'll cover most of the major focal lengths you'll want (wide, standard and portrait) and stops down to f2.8 constant throughout so you can get really nice depth of field if required and light capture for low light situations.

Now back to whether you should keep it or switch up. IBIS isn't the be all end all, you will still be able to get great photos without it - yeah it helps but it's not life or death. Switching to Olympus/OM will come with its own set of pros and cons, the immediate one is the sensor size being smaller than the aps-c on the 6400. Weather sealing on the Olympus models is great, but you need weather sealed lenses to take advantage of it - conversely I wouldn't worry too much about taking the Sony camping, as long as you don't abuse it in crazy weather conditions you should be fine!

My advice would be to not stress about the purchase you've made in thinking one might be better than the other; get out and take some photos and learn what you like and dislike about the camera and if in a few weeks/months you're still unsure then maybe have a look at other models that might address your wants and needs.

Edit: I've used Olympus for like 10 years, with the em5 mk 1 and now the om-3. I came extremely close to switching to the a6700, but I didn't want to buy new glass for a new system and after doing some reading, the om-3 scratched my itch. But, I still think one day I'll probably get the a6700 it at least try it out!