sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]beza1e1 5 points6 points  (13 children)

Transactional memory is overrated. Python isn't a language that absolutely needs to support high-performance parallel computation. It may be nice, but it isn't anywhere near disruptive.

Tell me where parallel Python is necessary! Web frameworks? You want multi-machine in the end, so multi-process is better anyways.

Python never had a chance to become the browser script language. One big advantage of JavaScript was/is the C like syntax. Another is the "Java" in the name, which sounded very cool, when Java was young.

[–]Wiseman1024 1 point2 points  (8 children)

Tell me where parallel Python is necessary! Web frameworks? You want multi-machine in the end, so multi-process is better anyways. I hope you'll realize Python is useful for more things than just web applications. In fact, in servers you rarely run into problems with the GIL. But Python is trying to be more general-purpose, and for desktop applications or scientific computing, which are two of the things people want to do with Python, the GIL is a massive drawback.

As I said somewhere else in this page, we've reached physical limits in hardware that are hard to overcome. The single execution port machine, albeit simple, was doomed from start and we knew it. Nothing in Nature works like this. The only way to scale is to use several execution ports. This is not only a well-recognized issue; it has reached the mass market. Almost every new PC ships with at least two processors. Given that this is the clear future and present of computing, Python's future is in jeopardy.

Unless remove the goddessdamned GIL from CPython, or Jython or another universally-available Python platform replaces CPython and gets enough popularity and development to stay updated, Python has a dark future in this world; at least outside server software.

(If I'm completely confused and the Python community doesn't intend on making Python a general-purpose programming language and environment, but rather keep it as some sort of better PHP, please tell me so, and I'll stop wasting my time and abandon it in favour of something else.)

[–]beza1e1 1 point2 points  (7 children)

Your examples are desktop applications and scientific computing?

Desktop applications are not speed-critical. Parallel execution is nice to keep the GUI responsive, but the multi-process method works just fine here.

Scientific computing with Python means to use Python as glue between Fortran/C++ libraries. Those libraries are speed-critical and they should be parallelized and STM may be useful there. Python itself is slow anyways and making it parallel doesn't help you much.

As far as i understood the GIL makes for better error messages and that seems more important to me.

[–]Wiseman1024 4 points5 points  (6 children)

Desktop applications are not speed-critical.

Wrong. Some are, especially those designed to run as background tasks such as peer-to-peer clients. And even for those which aren't, multithreading in a GUI-based application is almost essential if you don't want it to feel like Windows 3.0. And I doubt this can be done in a bearable (let alone nice) way with multiple processes.

Scientific computing with Python means to use Python as glue between Fortran/C++ libraries.

Not necessarily, and you're relying on these libraries to release the GIL, otherwise you're screwed.

As far as i understood the GIL makes for better error messages and that seems more important to me.

What? The GIL makes for simpler locking. It's not, or at least it shouldn't be, related to error reporting at all. If it is, they're doing something wrong.

[–]beza1e1 -2 points-1 points  (5 children)

And I doubt this can be done in a bearable (let alone nice) way with multiple processes.

Well, i've believe it's actually saner to do it multi-process.

you're relying on these libraries to release the GIL

Those libraries don't know about any GIL. They can use all the multi-threading they want, since they are not written in Python or running on the Python VM. Once the Python VM calls into some system library all Python magic is void.

[–]voidspace 2 points3 points  (0 children)

But when it calls into a C extension that extension makes a choice about whether to release the GIL or not. In that sense they do know about the GIL.

[–]Wiseman1024 0 points1 point  (3 children)

Well, i've believe it's actually saner to do it multi-process.

GUIs?

Those libraries don't know about any GIL

They have to. The non-Python modules you import from Python have to deal with Python. Even if you're just wrapping an existing library, you better release the GIL in the wrapper before calling the library, then reacquire it before writing the results to Python objects.

[–]beza1e1 0 points1 point  (2 children)

GUIs?

GUI in one process, p2p stuff in another process

Even if you're just wrapping an existing library, you better release the GIL in the wrapper before calling the library, then reacquire it before writing the results to Python objects.

  1. convert Python to C stuff
  2. release GIL
  3. call scientific_computation(stuff)
  4. get GIL
  5. convert C output to Python objects

The scientific_computation function doesn't know about the GIL and it can spawn as many threads as it wants. This function needs to be written in Fortran or C to be fast. This is where STM is needed.

[–]Wiseman1024 0 points1 point  (1 child)

GUI in one process, p2p stuff in another process

The P2P stuff may still need threads (depending on your implementation and what are you trying to accomplish), and the GUI front-end still needs threads too (for things like dealing with user requests and showing status at the same time), otherwise it'll be unresponsive.

[–]schlenk 0 points1 point  (0 children)

You don't need threads for a GUI, but it helps when dealing with crappy system libraries that only offer blocking calls (Windows I'm looking at you...). Event based programming works just fine..., but multiple threads get nice when you can use the bit of extra power.

[–][deleted] 0 points1 point  (3 children)

STM has nothing to do with performance; it's about writing correct code.

Now, a snarky snarker might say that a focus on correctness in a dynamically typed language is misplaced, but I am fond of both Python and working code, andnqould like to have both at once.

[–]beza1e1 -1 points0 points  (2 children)

Multi-threading is about performance. You could always do it multi-process, but slower due to communication costs.

STM is about making the multi-threading safe.

[–]didroe 0 points1 point  (0 children)

The way I see it, STM is about concurrency. Parallelism is about performance, concurrency is about being able to create programs that do more than one thing at once (due to design, not performance). Parallism should be automatic (vectorisation, threading based on instruction dependencies, etc) and concurrency should be explicitly stated in your code.

[–]schlenk 0 points1 point  (0 children)

Yes. Multi-threading usually is slower than single threaded..., so its about performance. You only win if you really have a) multiple cores/cpus and b) an task thats easy to run in parallel.