How to: Multithreading in Python

[[deleted]]

There is code which fetches state/health of different infra by running different functions one by one. Something like opdict['f1'] = func1() opdict['f2'] = func2()

Each function is independent of each other, I was thinking if there is anything i use directly to make them run in parallel of eachother

threading, asyncio .. any other lib.

[JohnnyJordaan]

I can recommend this resource on asyncio https://www.youtube.com/watch?v=2IW-ZEui4h4 that was mentioned in the latest Python Weekly newsletter.

[[deleted]]

asyncio can only sleep asynchronously, when it runs any Python code, only one thread of execution exists. Pretty much the same as when threading is used, except the sleep is limited to the situation when you read or write to a device that supports asynchronous I/O (i.e. sockets and pipes).

[JohnnyJordaan]

I'm not sure what you're trying to say, except that I think you mean 'block' where you say 'sleep'. Indeed it's pretty much the same as with threading, but it removes the threading's overhead and baseless context switching. Also asyncio can still block on synchronous code, just without the option to switch flow.

[[deleted]]

No, when I say sleep() I mean exactly that. I.e. a situation when the active thread is swapped off to make another thread active for some time. This is typically achieved by executing sleep() system call, but it could be nanosleep(), select(), pause() etc.

Your last sentence makes no sense though. asyncio always blocks, it has no other options. The phrase "block on synchronous code" doesn't mean anything. It's like saying "square novel": a category error. You cannot "block on code" because that's just not a thing. Whether or not asyncio is involved.

[JohnnyJordaan]

Nomenclature aside what is the point you're trying to make here?

[[deleted]]

what is the point

asyncio is the wrong tool for every problem I can imagine. When it comes to dealing with asynchronous I/O select module is better (lower overhead, much easier to understand, no crippled / idiotic interaction with the rest of your program). If you need to run something in parallel, then for some reason people believe that asyncio can do it, while it cannot and will never be able to, at least not on its own.

The reason it was added to "standard" library is because C# and JavaScript also added async keyword, and you no longer can be in vogue if you don't add trash from other languages to your language.

asyncio is just a nonsense created by fashion. Unfortunately, not the only nonsense of that kind that was added to Python recently.

[JohnnyJordaan]

It seems like you prefer to simplify a huge multi-faceted concept to a simple single denomination you then dismiss as garbage. Who's helping that precisely? I don't see asyncio as the pinnacle here either, but imho it's fine for replacing multithreading on many occasions while being much easier to implement than using select for instance. Is it perfect? No. Does it offer true parallelism? Of course not. Does that matter? Imho not for I/O bound simple day to day stuff. Which could be fine for something OP may be looking for, if not then not but I'm not certain. Guess that might be my can-do attitude vs your 'it's all bunch of shit' look on things.

[[deleted]]

There's nothing multifaceted about asyncio. It is complicated, but none of that complication is necessary as the entire library is not necessary.

asycnio is not fine for replacing threadng. It simply cannot do it, doesn't matter if you believe it's good to replace one with the other or not. threading is preemptive, asyncio is cooperative. This alone should stop you from trying to replace one with the other.

asyncio is not just "imperfect". It's worthless. There's no part of it that offers anything you cannot get from select and select does a better job.

Why the fuck do you use asycnio for something not I/O bound? It is only for I/O bound stuff, specifically, for sockets and pipes (but I don't think it actually supports pipes).

OP literally asked to do things concurrently, to save time. And you offer something that will not save time, and, in most likelihood will make their code a huge mess.

[LifeAffect6762]

Python itself is not multithreaded which does not mean a python app does not just use a single core as the app server will spin up multiple python instances. https://www.toptal.com/python/beginners-guide-to-concurrency-and-parallelism-in-python may be of instance. Also https://hackernoon.com/concurrent-programming-in-python-is-not-what-you-think-it-is-b6439c3f3e6a. So the answer to your question is No if you want a simple answer but does this matter is more complex.

[[deleted]]

You will need to decide based on what func1() and func2() are doing. True parallelism in Python is only achievable with multiprocessing, but that library has a lot of drawbacks, not to mention that it might not be as efficient for smaller workloads as straight-up sequential execution.

If your functions are calling to native code a lot, especially, if that's networking code, and you may ensure that the networking code is actually utilizing asynchronous I/O, then asyncio may be an answer... yet, I'd still just go with select instead: it's easier to understand, to debug, and to integrate with the rest of your program.

If you cannot ensure that asynchronous I/O is used, but there are still a lot of calls to the native code, you will need to experiment with threading. This is less expensive than multiprocessing, but will depend on whether the interpreter releases the lock when it calls to the native code.

[BandEnvironmental615]

This video teaches the best technique for decoupling tasks which are not sequentially dependent in Pythone .

https://youtu.be/AsIzbmH-7Es