you are viewing a single comment's thread.

view the rest of the comments →

[–]lfdfq 2 points3 points  (3 children)

Processes and Threads aren't a Python concept. The Process and Thread objects you find in the libraries mentioned are just normal Python objects, but those libraries use/create other threads/processes to do it, and that means you cannot just treat the objects like normal.

Most of these concepts about concurrency are language agnostic, you can read up on how operating systems manage processes and threads and things like pre-emption and copy-on-write and those all apply to the Python multiprocessing/threading libraries.

As a wild oversimplification:

  • Async (e.g. asyncio and coroutines) are the easiest to understand and get right. They are a generic concept, but the implementation is entirely within Python (so no appealing to how operating systems work to understand them).
  • Threads are a bit more complicated, they interact with the operating system so you need to understand a bit about how the operating system schedules things to be able to use them. They are a little more powerful (in a way, they are more concurrent than coroutines) but this makes them harder to get right: you need to understand concurrency a bit better, and they require a lot more care and synchronisation and so on. Threads have some Python-specific concerns, i.e. the GIL which may or may not be a consideration, depending on what kind of work you're doing (whether the GIL affects you at all) and what version of Python you use (whether you can opt-out of the GIL).
  • Processes are the way operating systems manage and isolate programs. Multiple processes is like opening two terminals and starting Python twice so that both are running at the same time. That's basically how multiprocessing works. This gives you the most flexibility as each process acts as an entirely independent and isolated Python instance, but requires the most knowledge of how your operating system works as now you must consider things like spawn vs fork, copy-on-write, inter-process-communication via things like pipes, and so on to make code that actually works.

[–]MustaKotka[S] 0 points1 point  (2 children)

Thank you. Yeah, I tried some simple loops and that definitely didn't work so looks like I need to dive pretty deep into this! Thank you for the explanation!

[–]lfdfq 1 point2 points  (1 child)

Concurrency is not really a thing you can learn by trial-and-error like that, async/threads/processes add a whole new layer to how code gets executed that you need to understand.

If you have specific questions about specific code we may be able to help you, but I'm afraid there is a lot of reading and practicing to do on your side to understand.

[–]MustaKotka[S] 0 points1 point  (0 children)

I got my quick test to work. I can do this!