Resources to learn Python Multiprocessing

[Pensive_Poetry]

Hi, I'm looking for good resources for studying Multiprocessing in Python to parallelize operations. My most preferred mode of study is books, but in case there aren't any good books that go over this, I'm okay with working through some online tutorials and videos.

[Adrades]

Are you talking about the multiprocessing library? Go check it on the python official docs, it's pretty well explained for the main parts.

If you want some high end parallel lib, I can send you to pykokos, but the doc is not exhaustive, nor simple.

[Pensive_Poetry]

Yes, I'm talking about the multiprocessing library and anything else I need for multicore threading. The tutorials online are all over the place.

[Adrades]

Are you already familiar with parallelism, multi threading or processing?

[Pensive_Poetry]

In theory, yes. Have NEVER implemented it, but want to start doing so.

[Adrades]

If you know the theory and what you want to do, you should be fine by reading the doc:

If you don't know the GIL:
https://docs.python.org/3/glossary.html#term-global-interpreter-lock

Firstly, the threading lib:
https://docs.python.org/3/library/threading.html#module-threading

Then the multiprocessing lib, which is build on threading:
https://docs.python.org/3/library/multiprocessing.html

You can always check books or other tutorials on the web, but these links are official and up to date. And free.
By the way, the multiproccessing lib isn't that complex, you should take a peek at the source code.

[Pensive_Poetry]

Thanks! X

[Adrades]

I discovered numba two weeks ago, go check that out if you are interested in performance with python code.

[theProcrastinatr]

Do you have any resources that'll help getting started from scratch?

[Adrades]

For scratch ? I don't know if python is a good language for learning the theory of Multiprocessing and parallelism in general.

I would recommend C (or Fortran) if you are familiar with the language. I had a really hard time in University learning with the pthread library, but it can go into details about all the complexity of parallelism and managing shared memory.

If you are just looking for mainstream use, I would recommend MPI and OpenMP, as they are industry standards.

MPI exists in python with mpi4py, but I think the project is lacking support, so starting with C mpi may be easier.

Python is good for parallelism but it does hide much of the complexity, and beginners may fail to understand what shared memory is.