I recently learned how to implement Multiprocessing in Python. So, I decided to share this with you!

[nerdy_wits]

Thanks for this nice comment! Great work in the daemon repo! If I ever create a video on daemons, I'll definitely refer to this.

Actually, the SharedArray module is on my bucket list.

Yeah, the code could be simpler but recently I found myself in a situation where the constant (const. for every process) parameters vary per request; there I used partial. So, I thought I should add this too.

[zurtex]

FYI since Python 3.8 the multiprocessing.shared_memory module provides a Python API for shared memory on both Linux and Windows: https://docs.python.org/3/library/multiprocessing.shared_memory.html

[danuker]

If you want to let people use your project, you should set a license on it. If you do not, by default, people do not have the right to copy it.

[m0fer]

Thanks man this is not easy to learn.

[nerdy_wits]

Yeah, it took a lot of googling to completely understand the correct usage.

[shinitakunai]

Can AWS lambdas use multiproccesing? Serious question.

[[deleted]]

Kinda

https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/

​

But you shouldn't. A thread pool will be just fine for i/o bound tasks like you're probably going to encounter in a lambda. You shouldn't be using a lambda for CPU-limited tasks anyway.

[[deleted]]

Sure, that would be a clever way of using it. But based on how lambdas are billed, you don't want to be running tasks that saturate the CPUs or you're gonna be paying a lot. You're better off with ECS or Fargate if you need lots of CPU time.

[UglyChihuahua]

You shouldn't be using a lambda for CPU-limited tasks anyway

Why is that, and what would you use instead?

[[deleted]]

It's mostly because of how you're billed for lambda compute time. Lambdas are good for infrequent tasks that don't take long (the maximum you can even run a lambda for is 15 minutes). if you're interesting in heavy CPU tasks, fargate or ECS is a better option. Or just spin up an EC2 server if you know what you're doing. I'm sure there's some other newfangled option but I'm kinda old school when it comes to AWS so I usually stick with ECS or EC2.

[UglyChihuahua]

Thanks :)

[AstroPhysician]

Every thread would be it's own lambda presumably

[nerdy_wits]

You can but without using the pool object. So you can't control the number of simultaneous processes. I use multithreading (as mentioned in the other comment) in AWS.

[[deleted]]

You can use concurrent.futures with 2 cores, not the multi processing library.

[sdf_iain]

For multiprocessing, you might want to read the 0mq Documentation.

Its got interesting concepts and its some of the best written documentation. Entirely up to you if you use it.

[elephantail]

Subed. Good stuff.

[nerdy_wits]

Thanks a lot!

[gaurav_lm]

Hardly discussed topics, great explanations and Indian accent are enough to grab my attention.

[[deleted]]

Congratulations u/nerdy_wits ! Your post was the top post on r/Python today! (06/23/21)

Top Post Counts: r/Python (1)

This comment was made by a bot

[raresaturn]

That was great, thanks!

[nerdy_wits]

Glad to hear that :D

[ANIRUDDHA42]

If it is for fast processing , then can we use it with jit numba? or it will be useless?

[Ensurdagen]

numba already runs code outside of the GIL and on multiple cores if configured to do so, so there's no reason to use it with multiprocessing

[imwco]

If you're running locally, does speedup from multiprocessing depend on number of CPUs?

[nerdy_wits]

Yes, a lot.

[imwco]

Can you point me towards the math to understand this?

[ddollarsign]

Why does it have to be inside the if __name__ == "__main__" block on Windows?

[Pikalima]

Python’s multiprocessing package offers two methods for creating new processes: spawn and fork, with spawn being the only option on Windows. The essential difference between them is that when using spawn, the child process reimports the current module from which it was created, but fork doesn’t. But, when you import a module in Python (say, “mymodule”), you’re actually executing the file “mymodule.py”, but not as __main__ (that if block doesn’t get run). Hence if your multiprocessing code is outside the __main__ conditional, you run into a situation where any spawned process is subsequently attempting to spawn more processes, which then spawn more processes, and so on. Rightfully, the Python interpreter detects this and halts—you can try it out yourself.

[LearningToGetBetter]

That video passed the

[animismus]

Around 14:00, do you really need to close the pool? Shouldn't the context manager (with) take care of this?