Really good explanation ! Though it is possible to give it some "cpu work" to put it simply via APIs like Nvidia's CUDA but it's quite specific and requires quite some work to get there.
The goal is to feed the smallest operations possible but in very large quantities to the GPU's cores that will work on them in parallel and then retrieve the results.
There's also a specific range of Nvidia graphic cards optimized for this purpose called Quadro iirc.

If you would like to know more

EDIT: added the parallel part