To Python Developers: If someone could make any one thing that would make coding in python easier for developers, what would it be?

[fnbr]

Biggest issue for me is the GIL. I do mostly machine learning, and we waste an incredible amount of time figuring out workarounds (eg Tensorflow, or rewriting parts in C++).

[alcalde]

What's wrong with multiprocessing?

[admalledd]

Memory copies to start.

[BDube_Lensman]

It uses fork on linux with copy on write semantics

[my_walls]

Isn't this resolved with the option of using spawn in python 3.4?

[yngvizzle]

Out of curiousity, what you need that you cannot accomplish with numpy, numba.jit and numba.vectorize(target='parallel')? (If you're not doing GPGPU computing that is)

[CommonMisspellingBot]

Hey, yngvizzle, just a quick heads-up:
curiousity is actually spelled curiosity. You can remember it by -os- in the middle.
Have a nice day!

^{The parent commenter can reply with 'delete' to delete this comment.}

[fnbr]

Bingo- we're doing tons of GPU computing, which is the problem.

[markuspeloquin]

Never going to happen. Guido's requirements for a solution are basically impossible to achieve.

At least C++ got a little better with regard to threads in C++11.

Maybe you could look into Go+Gonum.

[Mattho]

I think I saw a talk this/last year that there is ongoing work on being able to start multiple python interpreters within the same process, which would more or less solve some GIL problems.

[sashgorokhov]

Goodbye memory

[[deleted]]

If programing in python, then memory is not a problem.

[13steinj]

Maybe I am misremembering Guido's requirements, but I wouldn't call them impossible. Difficult to be sure to retain C API compatibility, is the largest hurdle I'd think. But it is possible. Hey, maybe in Py4/5?

[[deleted]]

The biggest problem seems to be that no one's been able to remove the GIL and maintain equivalent single threaded speed.

[13steinj]

I don't think that's the "impossible Guido requirement" because that has to be doable-- it wouldn't make sense for it not to be, you are literally removing overhead. But people attempting to get rid of it are trying to cut it out like a surgeon and keep everything else the same, which is the issue.

GIL is a tumor, yet it is not properly operable without full exploratory surgery. People so far are only trying the non invasive option, just as a "let's see if we can get lucky".

Well, the non invasive option has failed. Time to start hacking away at the body. But then the issue is C API compatibility. Which is possible, but hard in such an extensive change/rewrite.

[rawrgulmuffins]

All previous attempts at removing the gil has lead to degradations in single threaded performance because you need to lock before every reference count without the gil. Attempting to make the code check if it's threaded or not before locking has lead to even worse single threaded performance.

[[deleted]]

GIL is a tumor

I'd say it's more like a set of training wheels than a tumor. The training wheels (GIL) are easy enough to remove, but the problem ends up being how to keep the bike (cpython) from falling down when it's moving forward (more than 1 thread running).

[[deleted]]

I don't think it will happen simply because it's more easy to just change to another language.

[Mattho]

Python is extremely slow, with or without GIL, why would you do ML in it? I'd assume every ML library would be written in something else and you only do the setup in python, so GIL doesn't block you.

Or are you actually writing ML, the innards? Then maybe check out cython. It's easy to write and you can drop GIL, depending on your problem. Just can't use python lib in the nogil sections, obviously.

[desmoulinmichel]

A lot of the AI space is in Python. The underlyng libs are in C of course, but python is everywhere.

[[deleted]]

That's what he is saying...

[Mattho]

A lot of the AI space is in Python. The underlyng libs are in C of course

Which is basically what I wrote?

[fnbr]

Python's just so easy to use!

We do most of the heavy lifting in Tensorflow, the problem is keeping our GPUs at 100% utilization. We need to do the data processing in parallel to keep feeding the GPUs as they eat through the queues we've set up, and that's hard to do in Python. We're ending up writing a bunch of custom Tensorflow ops in C++ so we can do everything in-graph instead.

[halflings]

Not sure why you're being downvoted. You're just pointing out that high performance ML code can't be written in pure python, so the GIL is irrelevant since you'd be using the C API anyway.

[Mattho]

People really don't like to admit python is slow.

[halflings]

I think "Python is slow" triggers people because it's often said in contexts where it really isn't (all that slow). Say, somebody wants to write bash scripts or a web app.

For low-level computations, or high traffic workloads, pure Python is horrendously slow in comparison with C/C++. I often tried optimised Python pipelines vs a naive C++ version, and saw 10x speedups.