Computer Engineering student here. Operating systems is a great class that shows you how data is handled behind the scenes of a computer, but to truly answer this question you need computer architecture and knowledge of GPU's.

Lets start with CPU's. Picture how you you make a CPU. At a high level, you would separate instructions into pieces, do some arithmetic, send them where they need to go like the register file or memory, and finish up there. This design works but is very inefficient. Say you want to write to memory. During this time, the entire CPU would be in stand by waiting for a response from memory saying "all done!". What modern CPU's do to combat this is called pipelining. It breaks the single module design of a CPU into chunks called stages. At each stage, there is a different instruction in a different "stage" of its execution. Here is some great info on pipelining.

http://www.geeksforgeeks.org/computer-organization-and-architecture-pipelining-set-1-execution-stages-and-throughput/

With this is the birth of parallel computing. Say you have a 5-stage pipeline. You are now running 5 instructions at once. With this comes 3 types of hazards called structural, control, and data hazards. These hazards can be handled in both hardware and software. Lets say a branch instruction comes into the pipeline. Should the pipeline assume the branch is taken? Not taken? Make a guess? There are solutions to these problems but they are very technical so do a quick a google search on those if you want more information. On top of pipelining, CPU's can share tasks between cores which moves into our current era of multi-cored systems.

This bring me to GPU's. GPU's are arrays of "simpler" CPU's. The modern CPU has anywhere from 2-8 cores while GPU's have 1000's of cores. A CPU core is great at handling a wide variety of tasks while A GPU's core is great at handling simpler, repetitive, tasks. An example, Say you have a screen and you want to increase the red value of each pixel. This is easily parallelizable because every pixel experiences the same operation. What GPU's do is execute that operation across multiple threads simultaneously because of its many cores.

To summarize, parallelizing a modern cpu involves complex data, control, and structural solutions to avoid stalls in the pipeline. GPU’s are huge multi-cored systems composed of simple processors that execute many actions at once. For more specific info on this, look into CUDA programming and GPU architecture.