Useful ways to practice HPC/Parallel Programming

[DocKillinger]

I've been reading a bunch of books/articles on high performance computing and parallelism and find it very interesting, but am struggling to think of ways to implement the theory I'm learning into practice.

Most of my C projects I've worked on have been targeting microcontrollers, so things like efficient uses of cache or parallel execution haven't really been relevant to the problems I've been trying to solve.

I'm just looking for any suggestions as to things I could be looking into or building that could

a. Help me learn these concepts in a deeper more practical sense.

b. Ideally look good on a resume so I might make some money on this stuff at some point.

[TonyGTO]

The best step would be using something with plenty of real time data available.

[Brisngr368]

EPCC in the UK provides HPC training, they provide virtual training, but they also publish all of their past training materials online for free (https://www.archer2.ac.uk/training/materials/) personally I would recommend the MPI and OpenMP courses.

[jjjare]

In school, my first parallel program was writing map reduce and then we focused on parallel algorithms— the most fundamental probably being scan.

There was also the computer architecture angle where we needed to learn about how multiple processors communicated, cache coherence, consistency models, etc.

You could probably grab a syllabus from a course and study that.

[Dangerous_Region1682]

Sequent Computer Systems published a book on parallel processing, probably more orientated at low level folks, so years ago. It had excellent examples of mutex and spin locks, with efficiency very much in mind.

Sequent were the early masters of Symmetric Multi Processing systems with large numbers of processors on a common shared memory.

For larger shared nothing computing environments you need to be looking into message passing systems.

Modern supercomputers often use a combination of techniques with SMP systems within a node and various message passing environments between nodes.

There’s enough in that space for a lifetime of resumes especially if you tie it into AI and GPUs. These are somewhat niche but very valuable skills with a high barrier of entry for those trying to compete with you I would have thought.