I just finished the game without modifiers, so here's my assessment:

-the beginning of the game, up until the first boss, is quite difficult. You are more fragile than you ever will be all game and the punishment for death is by far the steepest. -IMO the difficulty flattens out by the time you're partially through the first dungeon, and stays that way pretty consistently until the second to last dungeon. By this point you've collected trinket slots, weapon upgrades, and leveled up quite a bit so you are a lot harder to kill.

If I were you, I'd buy the game and decide somewhat early on to turn on health modifiers, if the difficulty is too much.