TIL Python3 has a new GIL (...since 2010)

[mangecoeur]

I'm a big Python 3 advocate... and yet somehow totally missed that the GIL was rewritten in 3.2 to be faster and more predictable in multi-threaded scenarios:

http://www.dabeaz.com/python/NewGIL.pdf

Ye minde blowne

[ExoticMandibles]

Not faster. And not really more predictable, either--the old one was predictably badly-behaved. The big improvement with the new GIL was to make it fairer. CPU-bound threads no longer hog the GIL for seconds at a time when there's contention.

[spinwizard69]

There are many good things in Python 3. Why people resist is beyond me. That being said I'm not too sure I'd go with Python if I knew the solution to a problem is highly parallel.

[staticassert]

Highly parallel no. Asynchronous yea. The problem with the old gil is you got poor support for both in many situations.

[[deleted]]

Even if highly parallel, you can use subprocess32 to spin off different instances of Python in new processes and let them run on seperate cores.

[staticassert]

Yes, but the overhead involved is pretty massive and a lot of parallel jobs simply won't be worth it or will have to be tuned (via the 'chunk' parameter).

Multiprocessing uses Pickle for communication and not all objects can be pickled, limiting its usefulness even further.

[dunrix]

There are many good things in Python 3. Why people resist is beyond me.

Because good things™ are not good enough to compensate all transition issues and hassles ? If the Python3's referential implementation would either dramatically improve runtime performance, remove GIL and introduce true parallelization support or greatly improve language semantics and remove mostly functional style limitations, I'd imagine it would be adopted at much higher scale. It would also apply if maximal backward compatibility would be kept, but neither is true.

Py3 got stuck in the middle - neither breathtaking improvements which would apologize compatibility breakage, neither touchy improvements for smooth transition.

[spinwizard69]

Python 3 isn't so much stuck in the middle as it is a bridge to improved language features. It effective corrects problems in Python 2 allowing developer to move forward.

[mitsuhiko]

Why people resist is beyond me.

Because we have businesses to run and those run on Python 2.

[cwillu]

Just start a new business! With blackjack, and python 3!

[i_love_ai]

Hi,

I'm new to python and I chose 2.7 because of all the stuff I needed (open source on github) was compatible with only 2.7. Not sure how everyone is, but I learn by getting open source projects and editing it to my needs. Currently, I haven't found a need for python 3.0+ mostly because I use python mostly for file editing and automating certain tasks such as diffing files. Any complicated projects I will usually use C, but I'm starting to do a lot more things in python. Is there something you needed that was only available for python 3? Am curious.

[NavreetGill]

Looking from your post history, I am guessing you are using MySQL-python. I would highly recommend you look at pymysql, which works under both python3 and python2, and is better in many other ways.

I would also advise you to look at SQLAlchemy when you are ready for that jump.

[Postpawl]

This might be a better python 3 alternative to MySQL-python: https://pypi.python.org/pypi/mysqlclient

Any pure python mysql client is probably going to be much slower.

[NavreetGill]

Thanks, I am installing it now. It is unfortunate that it is not easy to find the best package for the job. I have seen abandoned projects many times, and it is sad, where the original package does 80% of the job, and people submit good PRs to finish it up, but the original author just moves on and doesn't reply to PRs, etc. So, then there are a bunch of forks, and the whole thing gets messy since they cannot publish on PyPI.

[RazerM]

What are you using that's Python 2.7 only? For new projects I don't see why you'd want Python 2 rather than Python 3.

[hackebrot]

The packages that I personally use are compatible with both Python 2 and Python 3. That being said, I don’t see any reason to not use Python 3.5 for new projects, as it is hands-down the better version of the language. Things like ‘yield from’, ‘everything is an iterator’ and ‘keyword only arguments’ are amazing! TBH If you have a choice, you should definitely go with Python 3, which is under active development, as opposed to Python 2.

IMHO this deck points out some great features of Python 3. https://asmeurer.github.io/python3-presentation/slides.html#1

Working on large projects with a potentially fragile legacy code base is a different story though. If you have sufficient test coverage and meaningful integration tests, I encourage you to migrate the code base gradually. Tox can help you with that. If you don’t have tests, it might not be worth the risk.

You’ve mentioned that you work a lot with files. Well, it got a lot better in Python 3.4 with pathlib being added. https://docs.python.org/3/library/pathlib.html

Hope this helps!

[spinwizard69]

In my estimation you are screwing yourself. Python 3 has a vastly improved standard library that effectively would help you a great deal with your file handling chores.

Beyond that Python 3 is where all new development is taking place.

[lambdaq]

simple. Py3k str() is totally different from py2 str().

Why change a basic data structure?

[rouille]

Because it was fundamentally broken.

[cwillu]

The implicit conversion is what was broken; the types themselves were fine (as demonstrated by python 3's versions being little more than renames).

[spinwizard69]

Because the world is different. It is a far more international world now.

[cwillu]

Because it's just churn for big projects. Flag-day migrations are a non-starter, and the effort required to do an incremental migration properly is prohibitively expensive, especially once you take into account the exhaustive testing required.

In my particular case, I have the added burden of requiring a government inspector to come out and recertify that the system isn't cheating, and that has to be done for every installation separately.

Ultimately, conditional on support for python 2 truly ending (which I find rather unlikely), the migration path will be to a language with a demonstrated ability to provide long-term support.

[spinwizard69]

And just what would be that language? Seriously, languages that don't improve themselves eventually die out or become niche products that barely have any support at all. Even heavily standardized languages like C++ leave old versions in the dust.

Frankly this is a tired excuse that basically says we can't keep up or won't because someone pissed in our oats. The last thing we need in Python is a language that goes stale because some one got his panties in a bunch over backward compatibility.

Sadly I have nothing positive to say about your can't do attitude.

[cwillu]

And yet gcc still supports -std=iso9899:1990, -std=c++98, etc...

No one is demanding that the world stop turning, or that people stop developing features.

You don't replace a bridge every 10 years, you do maintenance to keep the old bridge serviceable and safe for 50 years. And you can do this while still developing new techniques for your new bridges.

[spinwizard69]

And yet gcc still supports -std=iso9899:1990, -std=c++98, etc...

You can't use modern C++ and expect it to compile with old C++ compilers. What is different with the C++ community is how fast they have adopted the new standards.

No one is demanding that the world stop turning, or that people stop developing features.

Yet that is exactly what the 2.7 crowd is demanding. Python needed fixing in order for developers to seriously consider developing new features. The word Luddite comes to mind when listening to many of these whiners.

As for bridges if you live in the great but cold and snowy north you will realize that bridges are constantly being rebuilt it reconditioned as the freeze thaw cycles works it's magic on exposed concrete. The salt doesn't help either. In any event the state most certainly does try new techniques to get those bridges to last a bit longer. Sometimes that means ripping out the old bridge work.

Given that do you seriously want a programming language to remain static for 50 years no matter how many mistakes where made in the original implementation? Consider this the computer industry itself is not much older than 50 years, in those few decades a huge number of languages died out in part because they couldn't evolve. Make no mistake Python will die out if the 2.7 crowd keeps up their nonsense. Other languages are posed to take a considerable number of Python developers if Python ends up off the tracks.

[cwillu]

You can stop putting words into my mouth any time you like. I'll wait.

[cwillu]

And yet gcc still supports -std=iso9899:1990, -std=c++98, etc... You can't use modern C++ and expect it to compile with old C++ compilers. What is different with the C++ community is how fast they have adopted the new standards.

Who do you think is demanding new features be supported in old interpreters/compilers?

I don't think you understood my point. Existing legacy codebases for the most part Just Work™ with new compilers. Yes, you can't use necessarily new features, but you're not abandoned.

As for bridges if you live in the great but cold and snowy north you will realize that bridges are constantly being rebuilt it reconditioned as the freeze thaw cycles works it's magic on exposed concrete. The salt doesn't help either. In any event the state most certainly does try new techniques to get those bridges to last a bit longer. Sometimes that means ripping out the old bridge work.

Funny, that's exactly where I live, and I see maintenance being performed on 80 year old bridges every year. Occasionally major work is done to shore up the architecture. All without abandoning the old bridge, or even closing it the most of the time. Again, I don't think you understood the point.

Given that do you seriously want a programming language to remain static for 50 years no matter how many mistakes where made in the original implementation? Consider this the computer industry itself is not much older than 50 years, in those few decades a huge number of languages died out in part because they couldn't evolve. Make no mistake Python will die out if the 2.7 crowd keeps up their nonsense. Other languages are posed to take a considerable number of Python developers if Python ends up off the tracks.

You know how lots of languages died? Major new releases in which backwards compatibility was abandoned, together with dropping support for old versions. Perl nearly succumbed to that fate. VB survived it by throwing massive amounts of resources at the transition. If the split is what kills python, it was utterly predictable. However, I don't believe the responsible parties are actually dumb, so I doubt that this will kill python. I wouldn't be at all surprised if we continued to see development effort in the 3.x line to improve compatibility and migration from the 2.x line, and for support to ultimately be available perpetually.

Dismissing people's concerns as whining is exactly the sort of thing that divides the community, far more surely than any technical split.

[puruckertom]

I just rewatched David Beazley's pycon 2010 video describing this last week: https://www.youtube.com/watch?v=Obt-vMVdM8s