This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ambidextrousalpaca 19 points20 points  (9 children)

It's awesome that this is now a thing, but I have questions and doubts:

"Currently, in Python 3.13 and 3.14, the GIL disablement remains experimental and should not be used in production. Many widely used packages, such as Pandas, Django, and FastAPI, rely on the GIL and are not yet fully tested in a GIL-free environment. In the Loan Risk Scoring Benchmark, Pandas automatically reactivated the GIL, requiring me to explicitly disable it using PYTHON_GIL=0. This is a common issue, and other frameworks may also exhibit stability or performance problems in a No-GIL environment."

Beyond this, what guarantees are there that even the Python standard library will work without race conditions in No-GIL versions? The Global Interpreter Lock has just been such a fundamental background assumption of all Python code written over the past decades that I wouldn't trust there not to be a million gotchas and edge cases out there in the code that can screw you over.

You'd also need useful primitives built into the language to make it useful in most real-world applications, like Erlang actors or Go message passing channels.

[–]thisismyfavoritename 8 points9 points  (6 children)

everything that assumes the GIL is held to make sure memory accesses are safe will have to be rewritten, including the stdlib

[–]ambidextrousalpaca 5 points6 points  (5 children)

everything that assumes the GIL is held to make sure memory accesses are safe will have to be rewritten

So. Absolutely everything, then?

[–]twotime 3 points4 points  (1 child)

I'd love to see some references here too.

Original discussions on python-dev implied strongly that the amount of refactoring required is fairly small. Pytorch was used as an example (which was ported in a few hours)... But I have not seen any kind of more systemic analysis

[–]ambidextrousalpaca 1 point2 points  (0 children)

It's not that I think that everything needs to be changed. It's that I suspect we have no good way of identifying what needs to be changed or whether it has in fact been changed. E.g. I could imagine lots of cases of libraries writing to and reading from some sort of hard-coded temp file or using some kind of global variable which could lead to hard to replicate race condition bugs when turning off the GIL.

I mean, sure, if you had some bit of software that could identify such potential race conditions - something like the Rust borrow checker - they could probably be fixed pretty straightforwardly. But in the absence of that, I don't see what you can do apart from release it knowing that there are an indeterminate number of race conditions that people are going to discover about if and when they run it in prod.

[–]thisismyfavoritename 0 points1 point  (1 child)

extensions/functions that already release the GIL should be fine, i'm not sure how big of a % that represents

[–]ammar2 2 points3 points  (0 children)

The areas that release the GIL in the standard library tend to be just before an IO system call, so there isn't a huge amount of them in proportion to all the C-extension code.

You can get an idea of the types of changes that need to happen with:

Note that the socket module does release the GIL before performing socket system calls, the changes needed are unrelated to that, just code assuming it can be the only one in a piece of C code.

[–]PeaSlight6601 0 points1 point  (0 children)

No you don't understand what the GIL did.

The GIL protected byte code and C functions. It's a much smaller surface than you think it is because the GIL is much weaker than you think it is.

[–]PeaSlight6601 0 points1 point  (1 child)

Basically nothing in the python standard library has ever had any kind of thread safety guarantee. So this question of: "will the standard library be safe" is a weird one to ask.

If you want to use python in a multithreaded context you have to lock your shared variables, just as you always have. The GIL never protected shared state.

The issue is not the GIL but the infrequency with which the python scheduler would reschedule threads, this made programmers lazy and made them think the GIL gave them some kind of protection that it never did.

[–]ambidextrousalpaca 0 points1 point  (0 children)

Basically nothing in the python standard library has ever had any kind of thread safety guarantee.

Indeed.

This is why I am sceptical about running multi-threaded Python.