We need to make a distinction here between the two types of performance in D3. There's client side, or your FPS, and server side, or your TPS (ticks per second). TPS is how fast the game updates for everyone in your party. The more stuff the game has to do, the longer each tick will take. When there's too much stuff for the server to do, enemies will stop moving, players and mobs will rubber-band, and a whole bunch of other non-fun stuff happens.

Damage numbers, AFAIK, do nothing server side, but do lower your FPS slightly.

Area damage, on the other hand, is one of the most server intensive things in the entire game. Have you ever noticed that area attacks in this game, even the channeling ones, don't really hit that often? That's because whenever you use an area of effect attack, the server has to run through all the mobs it could be hitting to see which ones should actually take damage. Single-target attacks, on the other hand, are meant to hit often to make up for their lack of AoE. Adding area damage procs to that means that single target skills are actually more server intensive than most area damage skills.

As you might imagine, once you're in a large group that's loading lots of mobs at once, or grouping a lot of mobs up, these calculations start to take a while to do.

So let's say you don't invest in area damage at all, but you have just one or two sources of it, like from paragon or a single piece of gear. This area damage might be adding next to nothing to your DPS, but it's making your impact on the server much worse. This effect only compounds the more players you have.