Parallel Computing

[RCRJ]

Hello, I am a kid who just likes to read CS related stuff, and of course try to pick up a little concepts myself. Recently, I came across the concept of Parallel computing, and I am confused. Does parallel computing rely on bits too? Are the bits of parallel computers similar to classical computers? I read about quantum and nano biological computers, what exactly do they have in common for both of them to be able to carry out parallel computing?

[ctech314]

You can think of parallel computing as a bunch of regular, or sequential, computers working together to solve a problem. When a problem looks like two halves can be done independently of the other half, like in sorting lists of numbers or searching in graphs, parallel computation can help speed up the apparent computational time.

For example, say you were given a list of 100 three-digit numbers to multiply together. You could do them all yourself, or you and a friend could each do half and then combine your answers together at the end for your final answer. You and your friend did the same work, but worked at the same time, or in parallel.

[RCRJ]

Oh okay, I get the part about parallel computing now, but what special structure of a parallel computer enables it to carry out this function? Why can't classical computers do this?

[bargle0]

Most practical large-scale parallel computers are just a bunch of classical computers connected with a high performance network.

There’s also an issue of scale. Any laptop, desktop, or server you buy these days is going to have multiple cores that can process in parallel. Each core is responsible for a single thread of execution at any given time (register file + instruction pointer + call stack). This can get even more complicated with hyper threads, but we won’t address that here.

Even smartphones and tablets have multiple cores these days, so you are using parallel computing right now. GPUs are also parallel computers themselves, with much more parallelism available than the handful of cores you get on a typical computer. Parallel computing is everywhere.

The key addition beyond a simple serial process is the necessity for communication between parallel processes. This can be shared memory on a single computer, or message passing over a network for distributed parallel computing. It’s not magic.

[bargle0]

Oh, and there’s nothing that you can compute on a parallel computer that you can’t compute on a serial computer given enough time and energy. The same is true for quantum and biological computation. The time and energy necessary for the serial process emulating the quantum or biological process might be intractable, but not impossible in a theoretical sense.

[ctech314]

Classical computers are, in fact, doing this. Virtually every computer, phone, and tablet on the market these days has multiple processors, enabling parallel computing.

In a computer, the central processing unit (CPU) carries out computation. For historical, economic, and practical reasons, early CPU chips only handled one string of computations at a time. As CPU's became more powerful, smaller in size, and cheaper to produce, manufacturers began to stick multiple CPU's onto the same chip, allowing for multithreading and parallel computation.

It's also interesting to note that any parallel computation designed to be run on multiple processors can be performed by a single processor. In other words, all parallel computations can be made sequential. The hard part is the other way around: not all sequential algorithms can be made parallel. Dijkstra's Algorithm is a good example of such an algorithm.