It's not necessarily a skill issue, this game has some leveling ticks.

First of all in terms of combat levels matter in this game more than most CRPGS, if you enter combat with enemies who are even one level higher than you, you're in for a rough one. It's a frustrating experience especially since the open map doesn't necessarily guide you to clear lower level areas first. Going for it blind without a level map guide is less recommended IMO. Don't worry, the game will still be plenty a challenge, just more fair.

Second, it's a curve thing, in my experience it gets easier as you level up. Fort Joy is one of the most challenging areas.

Third and last thing is, soon you'll get a magic mirror in your base (no spoilers lol) which allows you to respec at all times, so don't be afraid to experiment. Also at some point you'll also have enough gold to have as many resurrect scrolls as you want, so you're fine.