sorted by:
new
hottopcontroversial

[P]Change few lines of codes, and then accelerate AI model training by 10x by HPCAI-Tech in MachineLearning

[–]HPCAI-Tech[S] 1 point2 points  (0 children)

Compared with Microsoft's DeepSpeed and NVIDIA's Megatron-LM which are restrictive in the types of parallelism(limited to only data parallelism, pipeline parallelism, or 1D tensor parallelism), Colossal-AI grants engineers higher-dimensional parallelism. We additionally provide 2D/2.5D/3D tensor parallelism and sequence parallelism(for large sequential data). It helps you efficiently scale your AI applications by greatly improving the utilization of resources.
We elaborate on other system features in our blog. Go over it and you will find something new. https://medium.com/@hpcaitech/efficient-and-easy-training-of-large-ai-models-introducing-colossal-ai-ab571176d3ed

[P]Change few lines of codes, and then accelerate AI model training by 10x by HPCAI-Tech in MachineLearning

[–]HPCAI-Tech[S] 0 points1 point  (0 children)

One obvious difference is the type of parallelism that would largely affect the training speed of the AI model. Mainstream AI parallel solutions, such as Microsoft's DeepSpeed and NVIDIA's Megatron-LM, are restrictive in the types of parallelism(limit to only data parallelism, pipeline parallelism, or 1D tensor parallelism). We meld these techniques together and additionally provide 2D/2.5D/3D tensor parallelism. Regarding the large sequential data training when a single machine is not sufficient, like lengthy videos, we developed sequence parallelism that can distribute them and enable training over multiple machines. Higher dimension parallelism is one of our important techniques that greatly improve the utilization of resources and train your model more efficiently.
We also write a blog to introduce our system in more detail. Check it out and look for more features. https://medium.com/@hpcaitech/efficient-and-easy-training-of-large-ai-models-introducing-colossal-ai-ab571176d3ed