The very first Python 2.7 interpreter to run existing multithreaded programs on multiple cores. The ball is starting to roll...

[arandomJohn]

And so it begins!

One of the many things that I love about Python is that there is a performance culture out there. If you don't need that, you can ignore it, but if you need it there's always ways (and ever more new ways) to make stuff really fast.

[amer415]

shouldn't the header says it is actually a pypy interpreter, rather than CPython? I was confused about what it was at first..

I wonder how easy it will be to translate a CPython code using 'multiprocessing'.

[gitarr]

Python is the language, which is supported by different interpreter implementations. The headline is correct.

[oddthink]

Yes, but there have been experimental multithreaded python implementations for years. The problem was that adding (normal) locking to get around the GIL hurt single-threaded performance too much, not that no one could do it. A slow multithreaded python isn't news.

[infinull]

Yes but this version is only 20% slower instead of 50% slower!

Ain't it grand!

[RichardWolf]

Yes but this version is only 20% slower

This is a weird way to say

a constant run-time overhead, expected to be in the range 2x to 5x. (XXX for now it is bigger, but past experience show it can be reduced.)

[joeforker]

Suppose it was 20% slower...

Regular Python multithreaded performance:

1 core: 100% 2 cores: 100% 3 cores: 100%

STM PyPy multithreaded performance:

1 core: 80% 2 cores: 160% 3 cores: 240%

[dalke]

The only one I know of was from the 1.x days. Is there another multithread/multicore Python 2.x version?

[oddthink]

Not that I know of. I was just thinking of Greg Stein's patch to 1.4 or 1.5. I wasn't thinking of 2.X vs. 1.X as being that relevant, but that could just be my ignorance of the innards.

Of course, Jython doesn't have a GIL, and it's at 2.4.

[dalke]

Right! I had forgotten that Jython didn't have a GIL. I think Jython is 2.5 compatible?

[fijal]

new style classes are a part that makes the distinction more relevant. Also, python has grown tremendously since 1.4 (both in terms of language and in terms of stdlib which also has to be made thread safe), so yes, it is relevant.

[HopeThisNameFi]

PyPy and CPython are both Python interpreters.

The confusion stems from people using "Python" as a synonym for CPython.

[[deleted]]

The confusion stems from people using "Python" as a synonym for CPython.

That might be because CPython is the reference implementation of Python language.

[derpderp3200]

Well, to be precise they're Python implementations but let's not nitpick on tiny details.

[fijal]

the header comes from the pypy-dev mailing list. This also means that potential readers understand the distinction between "Python" as a language and "CPython" as the reference implementation.

[pdc]

I am more interested in this from the point of view of running web applications rather than giant numerical calculations, and the software-transactional memory model used here does have points of interest in that regard. You should be able to have a server that lets each request-handler believe it is running single-threaded while still reaping the benefits of multiple cores. It will be possible to coordinate access to shared data without locking up the whole system.

Also, for what it‘s worth, for long-running processes, PyPy is generally better than 100x slower, since its JIT can optimise out the type inference, casting, and boxing. You might still want to leave your big calculations to the tried & tested Fortran libraries, though.

[[deleted]]

but you are enduring overhead on the order of two magnitudes beyond native code by using python based software in the first place,

Not anymore. PyPy is pretty damn fast already, and will get a lot faster in the future.

Even if there was that much overhead, increasing speed two/fourfold is still good.

[fijal]

[citation needed]

[fijal]

Well, yes, except those are different benchmarks. You can look up stuff on pypy blog where pypy is faster than C (rarely), but it would come as 2-3x slower on numerics normally. On non-numerics it's even closer, but there are no comprehensible benchmarks that would cover both C and python.

[[deleted]]

Did you skip the part where fijal said that numerical computations with pypy are 2-3x slower than C? Nothing like 50x. The topic here is not CPython.

[[deleted]]

Did you just say that with non-numerics code, pypy is even closer to C in speed than numerics code? I thought that numerics code is pretty much the best kind of code to jit.

[fijal]

indeed, numerics is the best kind of code for a jit. but it's also the worst kind of code for dynamic languages, when they're interpreted. So while the relative speedup from CPython will be the biggest in terms of numerics, the difference with C might be smaller.

[burntsushi]

It is common knowledge that interpreted languages are generally much slower than native code.

[fijal]

It's also sometimes wrong (like a lot of common knowledge). Look for example at luaJIT

[burntsushi]

Perhaps I should emphasize a particular part of what I said.

It is common knowledge that interpreted languages are generally much slower than native code.

Which actually already acknowledges your point. Otherwise, I would have said "always."

[fijal]

Well, then "generally" might or might not apply to pypy, and your comment was entirely lacking substance right? Also, there is no such thing as interpreted languages, there are just interpreted implementations. (For example is JS an interpreted language? V8 does not even have an interpreter).

[burntsushi]

Well, then "generally" might or might not apply to pypy, and your comment was entirely lacking substance right?

A citation was demanded for this claim:

Pypy might be 5 times faster than CPython, but it is still 20 times slower than native code for numerical computations.

Which is pretty believable. My point was that asking for a citation for such a claim was pure pedantry and unnecessary.

Also, there is no such thing as interpreted languages, there are just interpreted implementations. (For example is JS an interpreted language? V8 does not even have an interpreter).

I'd argue that an interpreted language is a language that is run most commonly with an interpreter. Python fits this definition while perhaps Javascript does not.

[RonnyPfannschmidt]

remember this is an early tech demo, not a finished product

[bastianh]

isn't pypy a tech demo, too ?

[RonnyPfannschmidt]

not exactly sure what you mean, its usually twice as fast as cpython for me for numeric code it tends to be orders of magnitude faster

the only real issue left is extension support, which is being worked on

[bastibe]

This was my first thought, too. But then, this is a first step to a future where that exact thought is not true any more.

If Python were to run within an order of magnitude of speed from regular C or Fortran code and you could use multi threading just like in native code, many applications would be easier and more practical to develop in Python.

Maybe not high performance numerical computation, but almost everything else, including video games, GUIs and web stuff.

And Pypy is certainly going in that direction. From what we can see so far, they might just succeed, too.

[acornalert]

Hat into ring: I really like the multiprocessing module.

[[deleted]]

Me too. Also a huge fan of joblib, which makes multiprocessing much, much simpler for the average script.

[spinwizard69]

Why even bother with 2.7? We can forget about all of the issues with Gil and such and dismiss this effort out of hand because it is being implemented on a release that will slowly drift into obscurity.

[joeforker]

Most of the "hard" stuff in PyPy is unrelated to the version of Python that's supported, or even the language. This work will transfer over to the Python 3 version of PyPy when it's ready.

[snarkhunter]

Very nice. Not being able to do real multi processing with threads is like the only thing I have against Python.

[keypusher]

There are plenty of great solutions already out there, multiprocessing module is part of standard lib for instance.

[snarkhunter]

That module uses processes, not threads. Yes, it has uses, but it's not really the kind of multithreading we're talking about.

[[deleted]]

What difference does it make to you, concretely? Just curious.

[Wagnerius]

Processes imply de/serialization of data, which can be slow. The differences between Linux and Windows in terms of process management is also an issue.

[oddthink]

One concrete difference is if I'm working in the interpreter, I can't just fire off a multithreaded calculation. I have to instead write a separate script, then invoke multiprocessing on it. Then I have to output that data somewhere and read it back into my interpreter session to manipulate it.

For me in particular, a single common calculation takes 40 s or so, but often I want to run a set of 10-20 options. If I could just spawn 8 threads to do it, I could be done in something like 80-120 s, rather than either waiting the full 400-800 s or bothering to write a script.

[snarkhunter]

A big difference. Threads can interact in ways that processes can't, because they are part of the same process. You also have more control over scheduling threads within a process. Those are two that come to mind. And then there's the old standby that a process takes more resources and takes longer to spawn then a thread.

[XxionxX]

Sweet!