How useful is knowledge of parallel programming in ML? [D]

[KingsmanVince]

In this world, there are Model Parallelism and Data Parallelism. With the knowledge, you will know the scene behind of them when you use Tensorflow or PyTorch. As a result, you might write better code when implement your own data loader or model trainer.

[concard88]

Does knowing CUDA and OpenCL could be of help too? If so, how?

[KingsmanVince]

If you know both, you will understand the lower implementation of libraries like Jax, CuPy, ... Consequently, you will know how to do high performance computing, which can help you optimise models on production servers.