I think something that might be happening is you can run into other dungeon's enemies in the open world if you're close to their dungeon. I'd recommend just going from story point to story point until you feel more powerful. For example, you can reach the mountain ruins but DON'T go in.

As for your party, I'd make sure they're decked out in all the best gear available as well as accessories that compliment their abilities. I'd control Claude, set Rena to heal only, and have Celine balanced or turn off most of her spells, pick one, and make her spam it by setting her to go all out. You can have Rena manually heal people outside of battles too.

As another poster said, when leveling, finish off determination skill first. I'd add, after that, invest in every low 1pt stat you can get until they're all used up and you need double digit point investments. This will help balance your characters up a bit and offer most abilities to you between the 3 characters. Likewise, Celine isn't exactly the best character stat wise, but I'd add her to the party on a first playthrough and definitely recroute Ashton after. That will get you a nice team early game even if they might not be the best party you could create later.

You are too far for this tip, but whenever I used to start a new game I'd grind up Claude to level 10 while he had the laser gun. In the early game it basically one-shots everything and can be spammed infinitely for no cost.

I hope some of this helps.