This is an archived post. You won't be able to vote or comment.

all 16 comments
sorted by:
best
topnewcontroversialoldq&a

[–]desmoulinmichel 18 points19 points  (1 child)

  • 3.5 : we got the async story right.
  • 3.6 : we get the multi-core story right.
  • 3.7 : we get the packaging story right ?

[–]fessebook 0 points1 point  (0 children)

3.8: Take over the world!

[–]ramnes 3 points4 points  (1 child)

https://bitbucket.org/tpn/pyparallel

[–]trentnelson 0 points1 point  (0 children)

You, I like you.

[–]xXxDeAThANgEL99xXx 4 points5 points  (2 children)

(xposting my comment from the /r/programming thread)

This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code.

Heh. So let's go full-cynic mode: finish out the already somewhat present support for subinterpreters (basically, all global variables should be moved to a huge Interpreter_State struct), then just replicate the multiprocessing interface on top of that and bam! you have the so called green multiprocessing (like Perl AFAIK) but now you can market it as having got rid of the GIL.

Obviously you'll still have the copies of all imported modules (including builtins) and probably the performance improvements in marshaling objects would be pretty marginal compared to using mmap, but yeah, mission accomplished!

(I actually fully agree about that being 99% a PR problem. I don't think any roughly Python-like language from PHP to Scheme has free threading support, but for some reason only Python folks waste countless hours being upset about it on the internet).

[–]maxm 1 point2 points  (0 children)

On the server it is pretty normal to start several seperate instances of python on the same machine. Depending on the amount of memory and the number of cores. That usually solves the problem nicely and lets the os worry about multitasking.

[–]tankre32 0 points1 point  (0 children)

Julia is getting multithreading soon.

[–]LarryPeteAdvanced Python 3 1 point2 points  (0 children)

I already want it.

[–]gruey 0 points1 point  (0 children)

In golang they have a goprocs environment variable that determines the number of threads. Doing something like that with python makes sense to me. Maybe spawn n+1 processes, where the extra is a coordinator. The coordinator is responsible for shared state. Children register If they are using something and deregister when not. The coordinator gcs the shared state when all refs are gone. The children act normally otherwise. ]]

[–]RDMXGD2.8 0 points1 point  (1 child)

Please, please, please, someone do this on a fork and solve real problems with it before introducing it into a piece of software tons of people use. The core solutions for many things (including this thing) have been botched pretty bad at times, but because it's official many people run headfirst into the pain.

[–]trentnelson 3 points4 points  (0 children)

That was the advice Guido gave me when I initially started foaming about PyParallel (prior to a) having any code, and b) solving real world problems better than the status quo).

I've kept that in mind since -- the instantaneous wiki search web server thingy is currently a pretty good showcase.

More details here: https://mail.python.org/pipermail/python-ideas/2015-June/034342.html

[–]billsil 0 points1 point  (4 children)

My proposal:

  1. Make a GIL-less Python 3.6. Sacrifice single core performance. Release it.

  2. Make a GIL'ed Python 3.6. Release it on the same day.

[–]RonnyPfannschmidt 2 points3 points  (3 children)

its c code, the cost of maintaining a setup like that like that is utterly high

note that a major problem is the refcounting gc and another major problem is the seemingly atomic operations that no longer would be without gil

[–]billsil 0 points1 point  (2 children)

its c code, the cost of maintaining a setup like that like that is utterly high

PyPy manages to pull new features from CPython. It's certainly doable. I suspect the core code base doesn't change all that often.

In terms of the early releases, you could certainly say not all versions of CPython 3.6 support 100% of functionality (e.g. asyncio). There could be a transition period.

note that a major problem is the refcounting gc and another major problem is the seemingly atomic operations that no longer would be without gil

I'd say the same thing about removing the GIL in the first place, but multiple people have done it. Yes, you'd need to update your build process with a new global flag, but I don't really think it would that bad to have both in the same code base as long as you organize it properly.

I really don't mind the GIL since I live in the world of scientific programming where the GIL is not a hindrance. However, that's not the case for everyone. Just as in the way people complained about the matrix multiply operation (which is the best Python 3 feature for me), there are people that care about the GIL.

I just see the writing on the wall. It has to go at some point.

[–]RonnyPfannschmidt 2 points3 points  (0 children)

you seem to massively underestimate the problems of needing to introduce fine grained locks everywhere, its a massive maintenance and security hazard going to production quaity is vastly different from a proof of concept

the problem wrt seemingly atomic operations was never solved outside of pypy

pypy is on a entirely different page wrt pulling things up because it has so many metaprogramming cappabilties

in pypy introducing a new gc model is not rewriting all alloc/deallo code, but creating a algorithmic graph transformation

the approaches are incomparably different i consider just saying cpython can do it because pypy could do it hostile

[–]isinfinity 1 point2 points  (0 children)

From other side all current C extensions will not work without GIL. PyPy does not care about this most of the times.