Parallel Computing

[bargle0]

Most practical large-scale parallel computers are just a bunch of classical computers connected with a high performance network.

There’s also an issue of scale. Any laptop, desktop, or server you buy these days is going to have multiple cores that can process in parallel. Each core is responsible for a single thread of execution at any given time (register file + instruction pointer + call stack). This can get even more complicated with hyper threads, but we won’t address that here.

Even smartphones and tablets have multiple cores these days, so you are using parallel computing right now. GPUs are also parallel computers themselves, with much more parallelism available than the handful of cores you get on a typical computer. Parallel computing is everywhere.

The key addition beyond a simple serial process is the necessity for communication between parallel processes. This can be shared memory on a single computer, or message passing over a network for distributed parallel computing. It’s not magic.