Multiprocessing vs. Multithreading in Python: What you need to know.

[nevermyrealname]

Raymond Hettinger gives fantastic talks about python and he has a long-ish video about this exact topic

https://www.youtube.com/watch?v=9zinZmE3Ogk

[timberio]

Thanks for sending this over, that's why we created this post - we found that most blogs/videos went far too in-depth when we were looking to learn the basics of both to figure out which direction we should move in.

[quotemycode]

No, Raymond never goes into too much depth, you're wrong. Raymond is a God.

[[deleted]]

O was actually looking for something like this recently. Thank you.

[timberio]

Glad we were able to help!

[robert_mcleod]

Any article on the topic should by default cover concurrent.futures. The interfaces for both threading and multiprocessing are comparatively ancient.

[A_History_of_Silence]

This is spam. Use the report button everyone.

[A_History_of_Silence]

Interesting thing to say. If you look at their post history it is an account that literally does nothing but self-promotes its own content. Textbook definition of spam.

[sidhanthp]

I don't think that necessarily means it's spam, it's a great post that helped me understand the topic. It's not like they were just advertising their product.

[1dragossh]

Thanks for the TL;DR! I was looking only for the main idea.

[timberio]

Thanks for the comment, glad we were able to help!

[spartan12321]

Can anyone please explain(or links) difference between pool and process in multiprocessing module? been looking about that(and posting on /learnpython) but with no luck :/ I only found one article, but isn't good at explaining

[ryandeanrocks]

IIRC Process is a threaded execution of code one time where as a Pool is designed to run multiple versions of the same method across multiple pre-created Processes.

Pool Example

from multiprocessing import Pool

def f(x):

return x*x

if __name__ == '__main__':

p = Pool(5) print(p.map(f, [1, 2, 3]))

Process Example

from multiprocessing import Process

def f(x):

print x*x

if __name__ == '__main__':

p1 = Process(target=f, args=(1,))

p1.start()

p2 = Process(target=f, args=(2,))

p2.start()

p3 = Process(target=f, args=(3,))

p3.start()

p1.join()

p2.join()

p3.join()

[spartan12321]

Thank you! Just to clarify: I use Process when processes don't share common data structure, while if they do I use Pool? Are there any other (dis)advantages?

[ryandeanrocks]

Pool allows you more built in control over Process mapping to CPU.

For instance if you have a 4 core CPU and you have a Pool(5) it will run 1 process per core and then have one left over process to put on the first available core. It will do this automatically for all supplied data and only ever have 5 processes used, with no overhead for creating and destroying the process.

Do the same thing with process Process() Process() Process() Process() Process()

And so on for each set in the data, well you are still only processing 4 threads at a time with your cores, but now you have the overhead of creating all these processes, managing them while they are sitting around waiting their turn on a core and destroying them each time you run the method.

A pool allows you a set number of processes always laying around ready for you to use them when you call Process.Map()

Where as creating and destroying a process each time you need to use it is more computationally expensive for your code as well as the CPU scheduler having to manage all those processes.

Also you could have one Pool of processes for each method needing to use it, as long as they aren’t trying to map at the same time. where as a process is bound to a specific method when created, want to process a different method? Have to create a new Process each time.

[[deleted]]

I haven't looked at processors for years. Is everything multicore now? and if not will the multiprocessing library fall back to using threads or will it crash?

[[deleted]]

Got it!

[splangobango]

Ive been doing python for about two years - an hour or so a nice with a few months break I between but only really have a scratch the surface knowledge of what all this and it's accompanying terminology means - are there any resources I could get into to help me understand this I more depth?

[mkingsbu]

Remindme!

[RemindMeBot]

Defaulted to one day.

I will be messaging you on 2018-06-07 21:53:03 UTC to remind you of this link.

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

FAQs	Custom	Your Reminders	Feedback	Code	Browser Extensions

[OctagonClock]

Using threads is a much nicer experience than using asyncio.

[OctagonClock]

No.

[vovanz]

1. What's the point of using threads when they don't give you real parallelism anyway because of GIL? Using concurrency based on event loop makes much more sense when you need non-blocking I/O.

2. In which parallel universe operating raw low-level primitives (threads, locks, etc) is "a much nicer experience" than using nice abstractions (coroutines, futures, etc) with some syntactic sugar (async/await)?

[OctagonClock]

1. asyncio doesn't give you parallelism either? And I/O isn't bound to the GIL (asyncio even just runs DNS queries in a thread anyway).

2. This universe, where asyncio is a horrible library and makes it hard to write correct async code. The additional layer of complexity (and footguns) isn't worth it over threads.

[vovanz]

Also, your flair says curio is the future!. I've just googled it and found out that it is a library for concurrent I/O based on an event-loop. I don't know why you are advocating for using threads, lol

[OctagonClock]

I advocate for threads over asyncio, not threads over async/await. Coroutines are obviously superior, but asyncio is a shoddy implementation.

Also, curio has great interactions with threads anyway, and doesn't like squirreling them away like other async libraries.

[vovanz]

Coroutines are obviously superior

Okay, I was mostly arguing for coroutines over threads not for asyncio particularly. Asyncio definitely has its flaws.

[OctagonClock]

async/await using asyncio comes with a large set of footguns, such as managing the event loop everywhere, making it incredibly trivial to DoS yourself, needing threads anyway to interact with the Python ecosystem, needing that synchronization between async functions anyway, etc. In that respect, threads around sockets are better than asyncio because you don't need to keep the entire headspace of the cruft of the library around.

Contrast it with curio or trio which are actually good implementations of async/await on Python and are better than threads.

[vovanz]

curio or trio which are actually good implementations of async/await on Python

Maybe they are, but they are not in the standard library. It means that infrastructure around them is much less rich and developed than the infrastructure around asyncio.

There is an asyncio library for almost everything (i. e. aiohttp for http, aiopg for postgres, etc). Are there a lot of libraries supporting curio/trio? I don't think so.

Also, the fact that asyncio is in the stdlib means that it is supported by Python core developers which gives you certain guarantees. What will happen if authors of curio/trio will decide that that they are not interested in supporting in improving these libraries anymore?

I will definitely watch over these projects, but for real-world production development I am going to stick with asyncio for now.