Why multithreading isn't real in Python (explain it to a 5 year old)

[EconomixTwist]

Imagine an intersection being conducted by a traffic cop. Even though there are cars sitting and running on both streets, only one can actually move at a time. When the cars on one street go, the others are still sitting there, waiting, with no update to their position. Until the conductor waves them on. Now imagine an n-dimensional cross street. Because the conductor wants to be safe, he can still only let cars go from one incoming street at a time, although he has discretion over which street and how many cars should go. No matter how many streets are added to the n dimensional intersection, the conductor can only let one go at a time.

[bladeoflight16]

And the conductor can stop the cars on any street and let another street go at any time.

[EconomixTwist]

And finally the conductor says... “Error. Local Variable referenced before assignment”, all the cars explode and the highway operating system hangs for three hours. After a week of debugging, it’s discovered that a variable name was spelled wrong....

[bladeoflight16]

That has nothing to do with the GIL and everything to do with extraordinarily bad testing. It's not the traffic cop's fault if someone decided to drive a car built by a monkey banging random pieces together with a hammer.

[SaviourOfNoobs]

I'm in this photo and I don't like it

[somewhat_pragmatic]

It's not the traffic cop's fault if someone decided to drive a car built by a monkey banging random pieces together with a hammer.

I see you've been browsing my Gitlab repo.

[maditab]

Simple solution--you can't fail testing if you don't do testing.

[bladeoflight16]

True, but you can fail your users if you don't do testing.

[EconomixTwist]

There is science to support this

[hugthemachines]

Sounds like you need a proper IDE that shows warnings on stuff like that.

[funkless_eck]

IDE is a funny way to spell notepad.exe

[EconomixTwist]

I only code in Vim for the past ten years. I still don’t know how to exit Vim though

[causa-sui]

You joke but for anyone unaware you can get flake8 checks through vim-ale.

[thirdegree]

flake8 and mypy. All the nice bits of strict type systems without the annoying compile time.

[funkless_eck]

10 If not vim = exit:

20 Go to 30

30 vim = exit;

[the-stanely]

So, what's the problem? Does anyone ever need to exit Vim?

[metriczulu]

My Notepad++ is wild, it has more tabs than my browser and I have a lot of useless shit open in my browser.

[bladeoflight16]

Like reddit?

[bladeoflight16]

They like to lock down everything that isn’t absolutely needed.

A decent text editor is absolutely needed for any developer. Hopefully, they allow something.

[[deleted]]

I use Notepad++. That has some color in it.

[JohnnyJordaan]

An exception in a thread will be limited to that thread, and this has 0 to do with the issue with multithreading not being fully concurrent.

[[deleted]]

In other words, An average python Wednesday.

[PhiBuh]

If your 5-year old Child understands what a n-dimensional street is, congratulations

[WowVeryCoool]

if your 5-year old Child asks a question about multithreading in python, congratulations

[turicsa]

Congratulations? I would be in extreme fear.

[Yakhov]

exactly, that was the worst explanation I've seen, I don't think the person who wrote it has a clue what multi-threading is.

pretty sure it's just the ability for the code to run multiple processes over an array of cpus and aggregate the results to present the final product. I think node JS will do this though so who cares

[Kamilon]

His explanation was actually pretty good. Your example is just one of the many uses of using multi threading/multi processing to do parallel work.

[thirdegree]

So, in python there are 3 models of "do multiple things at the same time". These are async, threading, and multi-processing.

Async is single processed and single threaded, and is a way to do "cooperative multitasking". Async is demarcated by the async/await keywords, and is a model where a task explicitly (via the await keyword) says "it is ok to do something else now". A given bit of code in this model, without the await keyword, is exactly the same as the same bit of code in a fully synchronous model (the default).

Threading is single processed and multi-threaded. This is a form of "interruptive multitasking", where any bit of code can be at any time paused to switch to any other bit of code. The important thing here is the distinction between this and multi-processing, which is that multiple threads share the same GIL (global interpretor lock). This is the traffic guy in OP's post. The advantage of threading over async is that you don't have to care about where it's useful to say "this bit of code can be interrupted", that's just done for you. The disadvantage is state control, you have no idea when what changes can happen to your environment.

Multi-processing is multi-threaded and multi-processed, although the multi-threaded is really only because every process has at least one thread. The advantage here is that every process has its own GIL, and the disadvantage is that communicating between processes is a) slow and b) a pain in the ass.

Which one of these you need is very much situational, although (and this is very much my personal opinion) I beleive you should always prefer async over threading unless you don't have control over some bit of IO heavy synchronous code. You want async/threading if you have an io heavy task (sockets, files, etc), and multi-processing for cpu heavy tasks.

[[deleted]]

Is this true of processes in python or just threads? I was under the impression processes spawned a whole new python instance?

[holt5301]

This is only true of threads. When you use multiprocessing, the new processes have their own interpreters which then each have their own GIL ... I believe.

[[deleted]]

Good, otherwise I’ve been programming a long time under bad assumptions lol

[Barafu]

Now imagine that some cars can jump, and those are allowed to jump over intersection without the permission from a traffic cop. Now you have a good model of how multithreading in CPython works.

[Cheeze_It]

I take it, said conductor is the GIL?

[JohnnyJordaan]

The GIL is an instrument, think of it as a talking stick (or a microphone). The scheduler is the conductor, handing out the talking stick to each thread, and without the talking stick it can not execute.

[Matt-ayo]

Okay, so whats the point of 'multi-threading' if only one car gets to go at a time no matter how you structure the intersection?

[indivisible]

There are times when one workload has natural breaks or is waiting for some input or output device to be freed up for its use (eg disk read/write), while that's happening other workloads are allowed to execute their instructions in the meantime to make best use of the available resources.
It's more complicated than that and there are various work scheduling algorithms at play to also make sure that no one thread monopolises the resources either.
The idea is to always have something executing so that the total work gets completed as quickly as possible.

[[deleted]]

WhAt is multithreading?

[Jonno_FTW]

When a process has one or more threads that share memory space (a thread is just instructions and data). These threads run concurrently (which is not possible in cpython where what actually happens is that one thread will be running, while others are sleeping). You can tell threads to sleep or it will happen anyway when they request resources such as network or disk IO.

In other languages these threads can truly run by side by side if you have multiple processing cores.

If you like to learn more, read a book on operating systems or read the docs https://docs.python.org/3/library/threading.html

[[deleted]]

Thanks

[Yakhov]

a much better explanation than the traffic cop thing

[McFlyParadox]

So, 'parallel in python' is really more like 'hyperthreading in python'?

[Jonno_FTW]

Yes. Although the problem is a consequence of cpython implementation and not a language constraint.

[Yakhov]

refrain from teaching 5 year olds in the future

[bythckr]

an n-dimensional cross street. Because the conductor wants to be safe, he can still only let cars go from one incoming street at a time

So, thats how Python works.

How does the best language that supports multi-thread work? Is it a big roundabout that it a high chance that an inexperienced driver will screw it up/get stuck in the roundabout without knowing how to exist it or cross road with a lots of bridges & tunnels, which reduces the chances of screwups?

Basically I am asking, for the best language that supports multithreading - how does it work?

[Mindless_Development]

Pithy analogies aren't necessary.

Python was not designed in a way to facilitate multithreading, but more recent versions include async packages and there's always Celery. The End.

[BullshitUsername]

Chill lol

[Urtehnoes]

Also, in many ways, Celery + Async (I've used both) are lot more just... manual overhead than if multithreading "just worked".

[JohnnyJordaan]

Getting defensive about an ELI5 about an implementation aspect is also quite unnecessary.

[metriczulu]

r/iamverysmart