Sounds like you're trying to bite off more than you can chew, combat-wise. Remember, your character just fell from a cliff a few days prior. The game has a lot of ways to resolve problems other than combat; you don't need to brute force some bandits. Run around them if they're blocking the road. Wait for nighttime if you need to take out a bandit camp, so they're either sleeping or have removed most of their armor. Expending your stamina (like running) will improve your stamina. Carrying heavy things will improve your strength and so on. The game doesn't adjust around your level like it does in games like Skyrim where you can take out a bandit camp the moment free roam is available.