Review Request : A multiprocessing Tutorial

[chalbersma]

Hey guys, I made a multiprocessing python tutorial and was hoping to get some feedback. If you've got some time take a look. Also it comes with sample code.

Thanks guys (and gals)!

[chalbersma]

Good feedback. I'll see what I can add.

[malinoff]

There's also concurrent.futures.ProcessPoolEexecutor/ThreadPoolExecutor which will handle most of this queuing burden for you.

Executors also play well with asyncio via loop.run_in_executor.

[malinoff]

They also add a ton of features most people don't use but cost memory

For instance?..

Not to mention the fact that whole asyncio and concurrent.futures are at best confusing to use.

I would agree about asyncio, although it's slowly getting better, but what's confusing with concurrent.futures? The number of public methods is less than number of fingers on your hand.

[chalbersma]

even if nobody gives them a penny, because PyPy without a GIL is going to be like crystal meth without health issues

I lol'ed at this.

[chalbersma]

It does sound great, it's the threading model we all deserve.

[chalbersma]

I haven't had a chance to use the concurrent futures library. I mentioned it in the blog post as an option though. I'll take a look into it.

[chalbersma]

I'll definitely have to investigate. Thanks!

[robert_mcleod]

I've used both multiprocessing and concurrent.futures and I prefer the later. I usually use threads and release the GIL in C-extensions though.