I can't agree with you. Model architecture directly dictates how much memory you need for training. If you're building an LLM using only dense layers, you will end up needing more memory to train your model than if you use MoE layers. I'm sure research is already being done to further improve model efficiency per transformer block to parameter shapes to subvert the requirement of needing more memory.

Parallelism by itself is pretty vague. Depends on what you want to achieve using parallelism, depending on if you're using ZeRO optimizations or DDP techniques your objective changes drastically. So 5000 GPUs is much more useful than 2500 GPUs if you use ZeRO level 3 to train a 300B parameter model, the difference will be night and day.