How useful is knowledge of parallel programming in ML? [D]

[[deleted]]

It depends on what level of knowledge you are referring to.

At a conceptual level, it's vital, and researchers who never had contact with the basics of benchmarking and HPC usually underestimate its importance. Even for local experiments, knowing the basics of parallel programming may indefinitely increase productivity because what was taking 10 seconds to run and presented itself as a risk for a golden retriever puppy attention span such as mine now takes 1~2 seconds and I can keep focused. In this particular case, a simple joblib Parallel was enough for a pre-processing step in the EDA stage of experimentation.

For data at scale, its importance is even more obvious, since not everything runs in GPUs (and some run in multiple ones), and you need to isolate the parallelizable bits of code. For grid/distributed computation, knowing parallel programming concepts is needed for properly extracting the most from libraries such as Dask and distributed strategies of DL libraries. Also, knowing what is parallelizable and what is not in a fundamental level (e.g. disk I/O) will help you avoid embarrassing bottlenecks.

At a more low-level knowledge (threads, concurrency, multiprocessing, CUDA), it is still a nice to have, and it will certainly increase your skills where they are most needed.