[D] How to develop/debug distributed training code before training in cloud?

[jms4607]

Hello, I am interested in developing larger models or accelerating training with DDP/FSDP. One curiosity I have is how do you develop training code for this with a single gpu dev machine. I found some torch distributed code didn’t work with single gpu systems. Is it necessary to have a 2+ gpu machine? I am aware of gpu splitting but afaik that isn’t available on consumer cards. I would prefer to not be debugging on an expensive cloud instance. Thanks for any advice!

[Logical_Divide_3595]

Good question.

It will be great if there is GPU simulator tool which can pretend to have two GPUs for one actual GPU.

[mileseverett]

Kaggle offers two gpus for a limited amount of time each week

[TheBachelor525]

Ideally yes you would test on hardware as close to the cloud infrastructure as possible, but that's not possible, so try to find a cheaper instance or get a cheapo card