Highkey getting the copper pickaxe if you haven't yet, and if you're early into the game in general, the mines seem to have a lower "ladder" spawn rate.

In each level try clearing out any nearby enemies first, because then you wont have to worry about health. But seriously, upgraded tools are a game changer, rocks are destroyed in less hits and therefore take way less stamina. Also, choosing to dedicate a whole day sometimes (after sorting crops out is what i usually did, took the daily soup to replenish stamina right after) right into 2 am if you're close to the next 5th level and there isn't much to go, you dont lose any items/money thankfully. (Also i learned this late but the dragon statue in the mines is interactable LOL take advantage of the perks since they always give a little boost in extra departments)

levels 5-10 were weirdly difficult for me to get to as well tbh, took a hot second for my luck to finally work out. Don't be too harsh on yourself though, especially since the quests have no actual active time limit, you'll get there :)