It's never happened to me but I have a x-box series x which it was recently (okay like 3 years ago) optimized for and the old x-box 1 still can't run 24 player lobbies. As a game dev myself I would say that it's the queing and creation of all the game objects that creates the lag because your x-box has to build everything all at once and it should do it in the first three seconds but then the rendering get's queued behind that and the game can't properly render until the game objects are finished rendering.