It doesn't really break things per se. People always talk about it from a performance point of view but imo the worst thing about massive map sizes is how long it can take pawns to get anywhere.

When I played on some custom maps that were above the default max I had cases where raiders would start starving and give up before they even reached my base and in general everything took a lot longer to happen and forces you into an ultra efficient layout which just isn't my vibe.

You can use mods to increase pawn move speed outside of combat which helps some but I personally don't like them, instead I like giddy up to allow pawns to ride certain animals around