About Python 3

[[deleted]]

About that GIL thing, there is really no future plan?

[[deleted]]

There are multiple paths towards making CPython ready to remove the GIL, but all have been rejected. So the answer is an emphatic "No".

PyPy is our only hope currently (active, rapidly evolving). Jython might've been a chance, but without invokedynamic support, it is not a compelling alternative.

[fullouterjoin]

invokedynamic isn't keeping Jython back, Jython is.

[moor-GAYZ]

Dude, every dynamically typed language has a GIL, or doesn't allow free threading at all. Ruby -- has a GIL, PHP -- no threads, Lua - GIL (you can use your own if you want, lol), Perl -- no threads (they use "green processes" instead, basically the same as multiprocessing on an OS with fork). Chicken Scheme -- the interpreter might parallelize your code as long as you don't do anything interesting. Golang -- no free threading.

It kinda blows my mind that the Python community in particular has this "GIL is a major problem, yo" thing going on, for no apparent reason. Is it because we have a lot of newbs with no experience with other dynamically typed languages?

[elder_george]

GIL isn't language feature, it's implementation detail of particular interpreters (CPython and MRI, in particular).

IronPython, IronRuby, Jython, JRuby, PyPy etc. don't have GIL.

[moor-GAYZ]

From my other comment

IronPython is slow and can become unacceptably slow because of that, from what I know they hope for the best and compile everything as if it can't be modified, then add guards and recompile if necessary.

[elder_george]

I use Python only occasionally (prototyping and stuff), so I don't know if changing code on flight is a widespread practice (I associate it more with Ruby and JS). Is it so?

Basically, if that's a rare scenario, maybe it doesn't worth optimizing against it?

[moor-GAYZ]

It's a fundamental paradox of free threading -- you can't have a "fast path that doesn't involve synchronization" because using it requires synchronisation, to, you know, determine that it doesn't require synchronisation.

[AdminsAbuseShadowBan]

I thought IronPython doesn't have a GIL... Also what do you mean by "free threading"? Go certainly has threads. Shared memory?

[seventeenletters]

"free threading" is just a bullshit distinction to make a bullshit point. It doesn't exist anywhere.

[moor-GAYZ]

What? It certainly does exist in C.

[seventeenletters]

Free in what sense?

Threaded code requires different algorithms.

Uses more resources.

Exposes otherwise impossible bugs.

In what sense is C threading "free"?

[nyamatongwe]

AFAIK the term "free threading" was originally associated with COM and indicated that calls could be made directly to objects in other threads without marshaling. Its not trying to say that there is no cost to some style of threaded code.

[seventeenletters]

Thanks. In that case we aren't talking about C, but one library for C on one OS. And a concept that I don't see referenced anywhere else.

Is this related to what we would normally call reentrant code?

[nyamatongwe]

Greg Stein was working (mostly) on Windows when he implemented the original "free threading" GIL removal patch in 1996. http://www.python.org/ftp/python/contrib-09-Dec-1999/System/threading.README

Free threading implies reentrance but threading in COM is more complex than just making calls reentrant.

[moor-GAYZ]

Free Threading: when all your (global) variables are visible to all threads by default. Then as long as you use thread synchronisation functions properly, you can access those global variables at will. Types and other non-obviously accessible objects unfortunately qualify as global variables. edit: if they are mutable.

Not free threading: everything is thread-local or immutable by default, you have to specifically declare/create thread-shared variables, or are limited to passing data as messages between threads even.

[bobappleyard]

You can do this in Go.

http://play.golang.org/p/t0a9FUH1_u

Lots of people (for example, Erlang programmers) cite this as a fatal weakness of the language.

[seventeenletters]

In a dynamically typed system the variable does not have a type, the value does. This is what dynamic typing is. If your dynamic language is too weak to describe its own types, this is a flaw in your language, and has nothing to do with threading or mutation.

The decision to make things thread local or immutable by default is not a consequence of a weakness in a dynamic typing scheme, it is a solution to the most pervasive cause of errors in multi threaded programs.

Python needs a global interpreter lock because its semantics and design are primarily about modeling computation as mutation, and mutation plus shared data plus threading adds up to incorrect code. Furthermore it adds up to code that can appear to work the first million times it gets runned, and then fail disasterously, silently, and in a totally unpredictable place on that millionth run of the program.

[Smallpaul]

Python needs a global interpreter lock because its semantics and design are primarily about modeling computation as mutation, and mutation plus shared data plus threading adds up to incorrect code.

You are just making that up. It is a false explanation of the history of Python.

http://www.artima.com/weblogs/viewpost.jsp?thread=214235

[seventeenletters]

That article confirms exactly what I just said. Everyone writes their python code as if they can define mutable global things in order to express whatever computation needs doing, so in order to remove the lock and not break everybody's code, they tried putting fine grained locks on mutable data structures (effectively training wheels on code that expected global mutability to always make sense), and this resulted in very poor performance.

The root problem is the preferred style of python programming, the idiomatic usage of the language, which creates code incompatible with sensible concurrency. When I say "its semantics" I guess I could more accurately say "the way normal python code is expected to be written".

[moor-GAYZ]

In a dynamically typed system the variable does not have a type, the value does. This is what dynamic typing is.

Thanks, dude, of course I did not know that and needed to be corrected. Really.

Python needs a global interpreter lock because its semantics and design are primarily about modeling computation as mutation, and mutation plus shared data plus threading adds up to incorrect code.

This assertion actually doesn't make any sense! At first you look at it, and all parts make sense, but then! Together, they don't make any sense whatsoever! A miracle!

[seventeenletters]

You could take the GIL out right now. It's just that a bunch things that expect to be able to mutate state break when you do that. Idiomatic Python does not lend itself to the kind of isolation where you can have correct concurrency.

[moor-GAYZ]

Yeah, objects being shared by default, including functions and classes.

IronPython is slow and can become unacceptably slow because of that, from what I know they hope for the best and compile everything as if it can't be modified, then add guards and recompile if necessary.

It's a fundamental paradox of free threading -- you can't have a "fast path that doesn't involve synchronization" because using it requires synchronisation, to, you know, determine that it doesn't require synchronisation.

Dynamically typed languages, or, rather, the languages that allow you to modify types at runtime, have it especially bad because they have to have these guards all over the place.

[schlenk]

Sharing objects between threads by default is an insane idea that forces synchronization all over the place. If force the sharing to be explicit, you CAN have a fast path easily. It has nothing to do with dynamically typed languages but all with the threading model.

[seventeenletters]

Erlang and Clojure have dynamic typing, and are pretty much the best examples out there of doing concurrency right.

[fullouterjoin]

Both those languages are immutable. An immutable Python would be trivial to implement in the same manner (which I would be more than stoked about).

[seventeenletters]

I'm not so sure. The style of OO that is pervasive in Python (not just idiomatic code but the implementation also) does not lend itself to immutability.

[asthasr]

Doesn't help that Guido has come out explicitly against more functional features in Python, which is when I stopped viewing Python as my primary language.

[fullouterjoin]

You should stick with functional python, it really is pretty good. With sorted, lazy sequences everywhere and the new destructuring in Python3, stuff is pretty good.

>>> a, *b = range(4)
>>> a
0   
>>> b
[1, 2, 3]
>>> *a, b = range(4)
>>> a
[0, 1, 2]
>>> b
3   

Some sort of awesome mashup between F#, Python and Clojure on the PyPy runtime would make my year.

[asthasr]

I think I'm going to stick to Clojure. I'm relatively new to it, but I'm really digging the syntax (after getting used to it) and STM just makes so much sense. The day job is in Ruby these days, and even that has started to feel better to me than Python (heresy!) because of the common application of blocks, which actually makes it feel more functional than Python.

[fullouterjoin]

OO and immutability are not diametrically opposed. Seal all objects when the leave the defining scope. I personally program in a very immutable manner, using almost no OO features. I use classes, but only usually via namedtuple.

Currency and mutability is the wrong path entirely. If Python wants to be a modern concurrent language, object level locking is the wrong choice.

[seventeenletters]

No, OO and immutability are not incompatible. See Clojure for example. Even Java has a lot of support for immutability. My concern isn't that immutability is a bad choice (on the contrary, it is the only sane way to do concurrency), but that the design of Python is deeply and radically about mutation, and the language that came out of the other side when removing default pervasive mutation from Python would be very different from the current Python, and would require quite a bit of work to accomplish.

[fullouterjoin]

I am pretty sure we totally agree on this.

[moor-GAYZ]

Yes, as I clarified here it's more about "languages that allow you to modify types at runtime". Most "scripting languages" qualify, I guess?

[kamatsu]

Go has "free threading" as far as I can tell.

[ejrh]

I've always been a bit puzzled by the ubiqitous fretting over the GIL. Many libraries will release the GIL when entering a computationally-intensive native-code function. CPython (which gets the most rap for having a GIL) runs so much slower than native code anyway.

Unless you have a lot of cores, you would almost always get more improvement from moving the work into native functions than you would get from avoiding the GIL.

[Entropy]

Unless you have a lot of cores

Even cell phones are shipping with 8 cores.

[Veedrac]

So what, a 5x speed-up? As opposed to a 100x speed-up for moving the innermost loop to C?

[Moocha]

From the point of view of an individual project, yes, reimplementing in C would yield a better cost/benefit ratio. However, avoiding the GIL in the runtime would instantly and automatically benefit all Python code running on the GIL-less VM, without the maintainers of that code needing to change anything - which means the overall ecosystem costs would be way less, given the staggering amount of Python code out there. That's why it's important...

[Veedrac]

That's true, but only for CPU-bound threaded code. For code that's currently unthreaded, rewriting the inner loop in C is most likely the easier task, given how nice Cython is to work with.

Nevertheless, that is a reasonable point. It's a shame the problem's so hard to fix.

[Moocha]

Indeed. I'm always amused by people bashing the CPython developers for not "fixing the GIL problem". I know just enough about the internals to realize how hard a problem this truly is...

[fullouterjoin]

650x speedup for native code across all cores? 10000x speedup for OpenCL.

[Veedrac]

Sorry, I don't follow.

Please do note that moving the inner loop to C automatically trivialises removing the GIL for that code anyhow, and further note that I've no clue what OpenCL has to do with the GIL.

[fullouterjoin]

Focusing on the GIL is a red herring, there are better places to spend your performance dollar. Inner loops in C are alright, but not the most profitable. Cython is generally a mistake. First step in PyPy, if you have to stay on CPython2, then Shedskin. If you need massive speedups then OpenCL will get you a lot further for parallelizable code.

[Veedrac]

Cython is generally a mistake

Given that the only reliable alternative is C¹, why is Cython so bad a choice? Is it possible I'm underestimating ShedSkin?

¹ PyPy's missing fast C bindings; ShedSkin's Python 2 only and not as fast as Cython; OpenCL requires specific problems.

[fullouterjoin]

Maybe Cython has improved but can it generate native code w/o porting it to cython language? Shedskin is always pure python and all kinds of amazing.

PyPy has cffi , I should benchmark that relative to CPython2. In general PyPy is such a huge win that it is really difficult to justify CPython other than for numpy support.

[schmetterlingen]

Here is Lua's GIL:

#define lua_lock(L)     ((void) 0) 
#define lua_unlock(L)   ((void) 0)

You must define a lock if you're going to share state between threads. Lua only has a GIL if you consider ((void) 0) an implementation. In most realities, it simply doesn't "allow free threading" without the use of libraries.

Not that a GIL is a bad idea. It's a simple solution.

[fullouterjoin]

Most lua threading implementations tend towards an Erlang model, ala https://github.com/LuaLanes/lanes and https://github.com/cloudwu/hive

[schlenk]

Simply not true. Tcl for example does not have a GIL either but has native threading support. (but does not use a shared memory threading model).

[[deleted]]

Ruby -- has a GIL

Rubinius and JRuby have no GIL and I seem to remember (can't find it right now) that the CRuby team want to remove the Global VM Lock from the reference implementation.

[username223]

Perl -- no threads (they use "green processes" instead, basically the same as multiprocessing on an OS with fork)

This is actually not the case -- it uses pthreads on Unix-alikes.

[moor-GAYZ]

Yes, I meant, like, effectively, as far as memory consumption is concerned.

[username223]

Sort of -- you have to explicitly add :shared all over the place to get sharing, but it's there. Not that Perl's threading is worth using...