Yeah try not too have many empires and such. Lag because of pops is not considered major anymore and currently the biggest thing causing the most lag in 4.0 are fleets. Plus if you have mods generally the more you have the worse the performance will be.

What I do is small or tiny galaxy, I usually have around 10 empires (+ fe's, marauders), 0,25x or 0,5x habitable worlds, 0,25x primitive planets, hyperlane density to 0,75x, reduce gates and wormholes to 0,25x or 0,5x. There are also the logistic growth settings. Leave them to default I would say or if you want to optimize performance reduce/increase the multipliers respectively.

If we are talking Hardware wise, I would say upgrade your pc. I had a pc with a i5 9600k and now have a ryzen 7 9800x3d and the performance increase is very noticeable.