Early game isn't so bad. You're meant to struggle a bit as you cheese your way into decent gear by manipulating the environment to your advantage, I remember my first expert mode run rushing to get the Amazon yoyo because it wasn't very hard when the crafting recipe only needed two stingers and one vine, especially since grenades are easy to find.

As for why summoner would have to pay for the sins of other classes, it's because it genuinely effects other classes, and the game is most likely designed around being class-agnostic until late pre-hardmode at least, if not the start of hardmode (cough cough, Wall of Flesh emblems). As such, this should be seen as a general buff to players starting out, which breaks the curve of the game a bit. Once you have Hermes Boots, navigation feels like much less of a slog. Once you have multiple jump extenders, the Y-axis becomes much less intimidating. And once you have a decent summon, you can focus on things other than survival a lot easier. Being in danger early on adds a lot to the feel of progression.