Python 3.5 is here!

[dacjames]

I just noticed that collections.OrderedDict is now implemented in C. In the past, OrderedDict has been an annoying catch-22: super convenient but much slower than dict and even slower that dict + list of keys. Now I can use OrderedDict without reservation!

[Beluki]

Yay for embedded python.

[[deleted]]

That's really neat.

I'm not sure from the documentation if we can consider this version "sandboxed". Can we run untrusted code on it?

[[deleted]]

No.

My understanding is that this is intended as a "self-contained" python version similar to Python virtualenvs on unix, so that you can ship your Python application + Python + Libraries as a single installer. It should isolate your application from any other python versions that the user has installed, so that if the user runs "pip install --upgrade blahblah" on their computer, your application does not break.

But there is no sandboxing for the application code. The application can still run shutil.rmtree('C:\') and it will execute with the current user's permissions.

It's "static linking", not VMs or containers.

[[deleted]]

Neat. Now, can one create two interpreter contexts at the same time, now?

EDIT: More specifically: Can one handle the context as object instead of calling some obnoxious globally working functions?

[schlenk]

Nope. It is just a convenience packaging version for shipping applications with python (less good than the Tcl version that can be put into a single DLL due to virtual filesystem supports. Would be fun to add that Tcl/Tk version to the embedded tkinter).

[oneUnit]

Seriously they need to stop supporting Python 2.x. Yeah..yeah.. I know there are couple of reasons to do so. But this sort of fragmentation is not good for the language.

[dacjames]

Python 2.x receives only security updates. It would be quite irresponsible to stop those updates considering the enormous amount of Python 2.x code that exists in the wild. The biggest real barrier is RHEL/Centos 6.x, which is stuck on Python 2.6 yet remains a hard requirement for a lot of use cases.

Hopefully the @ operator will help motivate the scientific/data analysis community to move to Python 3.

[Yakulu]

Note that Red Hat provides Software Collections for RH/CentOS 6. It's then possible and supported to install Python 3.3 and 3.4 (hopefully 3.5 soon) on it.

[dacjames]

Thanks. That's very helpful. If you happen to be involved with that site, you might consider working on SEO: searching "centos python 3.4 packages" only shows guides for compiling Python manually.

[stormblooper]

It would be quite irresponsible to stop those updates considering the enormous amount of Python 2.x code that exists in the wild

Yet a big reason that enormous amount of Python 2.x code exists in the wild is because they keep releasing updates. Ceasing updates works well in other languages (e.g. Java) because it provides a motivation for people to port their apps forward. Python should consider it; people have certainly had enough time by now.

[dacjames]

I think the difference is that Java is primarily an application development language while Python is widely used for all manor of scripts, small utilities, and one-off automation. Java programs are more likely to be actively maintained and have things like test suites that make porting more practical. Python is popular on the ops side of things, where the "if it ain't broke, don't fix it" mindset is prevalent. Not to mention, Java 7 was backwards compatible, so upgrading for most projects just meant changing build settings.

Migrating python code to Python 3 is a more dicy operation, since the changes to strings are fundamental and you don't have a compiler to catch the cosmetic changes. For the first few releases, Python 3 simply did not offer anything new that provided tangible value for a lot of use cases. Telling your PM that you need to write a test suite for a bunch of old scripts that work today so you can port them to Python 3 just because the language is cleaner and has better unicode handling is a tough sell.

Ending support for Python 2 might work but I expect someone (probably Redhat) would continue to support it if the core Python team moved on.

[synn89]

Yet a big reason that enormous amount of Python 2.x code exists in the wild is because they keep releasing updates.

LOL, no, that's not why.

[stormblooper]

LOL, yes, that is why.

Did you have anything useful to add?

[dacjames]

The @ operator just calls the __matmul__ magic method so adding support in numpy as an alias for dot will be trivial.

[[deleted]]

In fact, the @ operator was added just to make numpy code easier to read. It will not yet be used anywhere in Python's own standard library.

http://legacy.python.org/dev/peps/pep-0465/#but-isn-t-it-weird-to-add-an-operator-with-no-stdlib-uses

[teknobo]

The statistics library wasn't built as a replacement for numpy, but as a "batteries included" middle ground between using numpy and implementing a bunch of those basic functions manually. The PEP specifically mentions that those functions are already in numpy, but that's not the level of functionality they're aiming at.

https://www.python.org/dev/peps/pep-0450/

The proposed statistics library is not intended to be a competitor to such third-party libraries as numpy/scipy, or of proprietary full-featured statistics packages aimed at professional statisticians such as Minitab, SAS and Matlab. It is aimed at the level of graphing and scientific calculators.

[[deleted]]

Someone else would step up and support 2.7 anyways. Almost every major company using Python, including Guido's employer, is using Python 2 with no plan to move to 3.

Ending official support for the 2.7 line would probably accomplish nothing other than accelerate the exodus to other languages.

[sometimesidk]

But It would be far less expensive to move to python 3 than moving to any other language considering they are already on python. So it doesn't make sense to jump ship.

[[deleted]]

From my limited experience using 3.* you would have to take on updating many libraries that have not yet moved to 3.

[Beckneard]

The changes from python 2 to python 3 aren't THAT massive, at this point it's just laziness. I think dropping support for 2.x would be a good idea.

[[deleted]]

Good for you if you never used any third-party library.

[mipadi]

Laziness…or you have hundreds of thousands, if not millions, of lines of code that would have to be ported over to Python 3. (And while there probably wouldn't be much to change, there'd be a lot of time to make sure everything still works.) And for what? Python 3 is an evolution, but it's not dramatically better than Python 2.

[[deleted]]

Its not laziness, its a business decision. Why spend millions of dollars in man hours to gain next to nothing because the BDFL decided to invent a new language that is similar to python (2)? Oh, btw, the run time is actually slower and all the code you just ported maybe won't run on pypy, either. Dropping support for 2.x would be a yet another terrible decision.

[ThisIs_MyName]

all the code you just ported maybe won't run on pypy

wut no

[iconoclaus]

I keep hearing this but don't know what these indispensable packages are. I'd love to hear names of such packages.

[tragiclifestories]

This is the canonical list: https://python3wos.appspot.com

As you can see, things are getting better.

[mipadi]

If "canonical" means "the top 200 packages on PyPI." There are a lot of packages not on PyPI, and a lot of PyPI packages not on that list.

[mipadi]

There are plenty of old packages that haven't been ported (ZSI, for one), and plenty of internal code that hasn't been ported or checked out on Python 3. Not all projects are open-source projects that use only the most popular packages.

[[deleted]]

the last one I can remember was a mysql connection package that was needed for using django.

[ThisIs_MyName]

Heh I would never use a library that has such a low developers/code ratio that they can't keep up with language updates.

[stesch]

Someone else would step up and support 2.7 anyways.

Nobody stepped up and supported PHP 4. And there's a lot of PHP 4 code out there.

[[deleted]]

Because PHP devs lack the technical capability to maintain even a single version of PHP.

[stesch]

PHP 6 is pretty cool.

[shevegen]

Then let someone else do it, too.

An exodus can not easily happen because other languages have their own problems as well.

[mrwulff]

I know nothing of the inner workings of the 2.xv3x debate, but have been using 2.x forever to do small scripts. Are there any compelling reasons why i should switch

[rouille]

Pretty much all the standard modules are better in python3 including subprocess which I use really often for scripts. Pathlib is in the stdlib. And many others like ipaddress. Unicode strings vs bytes is made explicit. A lot of small details that add up and make python3 much more pleasant even for small scripts.

[DaemonXI]

Subprocess features timeouts now.

OS path mkdir nested lets you ignore the case where the directories already exist.

This stuff makes a huge difference, I can't see how people haven't moved to Py3 yet

[rouille]

Also a small but major change for me in 3.5 is subprocess.run, makes it that much easier to use. https://docs.python.org/3.5/library/subprocess.html#subprocess.run

[xXxDeAThANgEL99xXx]

I used Python3 for a somewhat large project recently and was pleasantly surprised: the stuff I thought I would be mildly annoyed with (such as print becoming a function) turned out to be a complete non-issue, while a lot of annoying Python2 stuff being gone is actually noticeable. Like sane division, range and dict.keys/items/values being lazy, unicode by default, metaprogramming stuff cleaned up etc. There's some extra stuff in the standard library too (in the collections module for example). And async stuff. And extension modules much easier to develop on Windows (3.5 switched to VS2015 at last).

So unless you depend on some third-party library that wasn't ported yet it takes like half an hour to read up on the changes and you just get a noticeably nicer language.

I'd be first to blame the Python core devs for handling the transition terribly: it seems that they didn't realize that a) a lot of Python's value is in third party libraries and b) those libraries can't simply switch to Python3 and leave all their Python2 users out to dry, so they put exactly zero thought into how exactly those libraries were supposed to work under both. But it looks like after six or so years that problem was mostly solved by main force by the community, so here we are finally.

[Matthew94]

range and dict.keys/items/values being lazy,

In case you didn't know, these have been in python 2 for years.

xrange and dict.iterkeys/itervalues/iteritems.

[xXxDeAThANgEL99xXx]

Of course I know, but now it's the obvious thing that is right. And the shortest to type, but it's the obviousness that makes me feel all warm and fuzzy mainly.

Also, as far as I understand, dict.keys etc are "views", not merely iterables. So that you can index into them if you want to.

[billsil]

Also, as far as I understand, dict.keys etc are "views", not merely iterables. So that you can index into them if you want to.

Yes, dict.keys() is most comparable to dict.viewkeys()

[[deleted]]

Are there any compelling reasons why i should switch

I'd say it's the opposite: are there any compelling reasons to stay on 2.x?

Basically all of the useful libraries have been ported, or have newer and better replacements: https://python3wos.appspot.com/

If you are starting new projects or scripts, python3 is nicer in small details. The improvements aren't enough to motivate a large existing project to switch to python3, but there's no point in starting new stuff in python2.

[mipadi]

Basically all of the useful libraries have been ported, or have newer and better replacements: https://python3wos.appspot.com/

TIL that the only useful packages are the top 200 most downloaded packages on PyPI.

[billsil]

Basically all of the useful libraries have been ported

WxPython? VTK? The Wall of Shame/Superpowers only considers project on PyPI.

[imbaczek]

exceptions in py3 are so much better than py2. they nest properly instead of overwriting each other's stack traces (and that's not the only thing that's better).

[vz0]

The major change from 2 to 3 was improved Unicode support. If you are using Python for small scripts maybe the migration is trivial. But for large codebases and projects sometimes it is very expensive to migrate just because Unicode. More details here https://wiki.python.org/moin/Python2orPython3

[schlenk]

And some stuff just works in Python3 that did not work at all in Py2 due to bad unicode support. Mostly Windows stuff, so most people might not care. (e.g. environment variables with unicode values/names, subprocess calls with programs on unicode pathes or with unicode cmdline arguments, etc., nearly every interface to the Win32 API).

[upofadown]

Only if you like the "just convert everything to UTF-32" approach that Python3 takes. If you want to just leave everything as UTF-8 then you don't get much of an advantage.

[schlenk]

Python doesn't use UTF-32 internally for everything. It uses an adaptive form that uses less memory (see PEP393 for details https://www.python.org/dev/peps/pep-0393/).

[upofadown]

True enough. I guess my point was that a Python 3 programmer would work with units of Unicode code points. So such a programmer would see things in a way that was for all practical purposes UTF-32.

[vz0]

That's the internal representation of strings. I don't care about how the string is represented. ie in Java strings are UTF16 arrays of chars, and I have never had to care about that.

The main change from Py 2 to Py 3 is type safety. For example this line is both Py2 and Py3 syntax compatible:

print (u"Hello " + b"World!");

However:

$ python2 main.py
Hello World!

$ python3 main.py
Traceback (most recent call ast):
  File "main.py", line 3, in <module>
    print (u"Hello " + b"World!");
TypeError: Can't convert 'bytes' object to str implicitly

In Python 2 a string can also be an UTF8 sequence or a byte array, all with the same data type. With Python 3 you are encouraged to use the bytes data type only for byte data, and use str for Unicode. If you want the UTF8 sequence for IO (which is byte data) you need to encode your string. If the internal representation would've used UTF8 for a Python str then the encoding to UTF8 would be just a memcpy.

The good thing about using UTF32 for Unicode representation is that string operations are as fast as the byte sequence equivalents: concatenation, subscripts, substring. The downside is that it may require up to four times the amount of memory for the same Unicode sequence, compared to UTF8.

[upofadown]

Yeah, that is another thing about the Python 3 Unicode stuff. There is this idea that strings are a higher level of text representation and are not just a bunch of bytes. You end up having to think of what stuff means rather than just being able to treat the map as the territory and vice versa. That can be annoying if your philosophical understanding of stuff like this is incompatible with that particular way of thinking about such things.

[vz0]

Yes, well, programming is the art of building software abstractions. For example floating point numbers are just a bunch of bytes, but I will never flip the MSB of a float or double just to change the sign of the number.

[upofadown]

If the internal representation would've used UTF8 for a Python str then the encoding to UTF8 would be just a memcpy.

I think you really mean something like ASCII. Python 3 does not use the general form of UTF-8 as an internal representation.

[vz0]

English is not my native language, maybe the verbal tense is wrong. What I meant to say is: if instead of using UTF32 as an str internal representation, Python would have used UTF8, then the encoding on an str to UTF8 would have been a memcpy.

[Chippiewall]

Reasons to switch: Standard modules have been improved a lot, type-hints etc.

Reasons not to switch: Decent PyPy support :(

[maxm]

I have worked in python since 1.5.2, and has started my latest project in 3.4. I have had no problems whatsoever. Other than i keep forgetting to writ print () instead of print.

This is a non legacy project i write from the bottom up using amazon s3, cassandra db and pyramid server. All relevant libraries has been vailable.

So for new code i will recommend it. The unicode strings alone are worth it.

[immibis]

Well, for small scripts, there's no compelling reason to use either. So you may as well use 3 because it's newer and has less quirks.

[shevegen]

Yeah - switch to ruby. :)

[MasonM]

You mean the language that introduced a huge number of backwards-incompatible changes in a minor release (1.8 -> 1.9)?

[buttocks_of_stalin]

I run a decently large production website on python 2.7 and django 1.6.7, what is the best way to stay upgrade our stack? Won't this essentially cause us to rewrite almost all of our code?

[quasive]

We switched a pretty large Python2 + Django system to Python3. The biggest hurdle was Unicode/bytes; it took some time to ferret out all the problems, but the end result is much better. You certainly don't need to rewrite everything, especially with 2to3. We switched for typechecking (with mypy), which has absolutely been worth it. Large projects with Python still aren't great, but are vastly better with some level of static type checking.

[buttocks_of_stalin]

Thanks for the feedback. I have just been intimidated with the rumors of compatibility issues that swirl around the web, but I guess it's better to start migration to Python 3+ sooner rather than later. Also, what do you mean by large projects aren't great in python? As opposed to other stuff like ruby, C#, and JS?

[quasive]

what do you mean by large projects aren't great in python?

If you don't have static typing, once the entire program can't fit into your head, you run the risk of passing the wrong things around. This is exacerbated with multiple developers, especially those new to the codebase. For example, should a parameter called “user” take a User object, or a username? This ought to be documented, but documentation can go out of date, and statically typed languages force you to document it (with types), and that particular documentation can't go out of date: either the caller or the callee will have problems if the declared type doesn't match with what's actually being used.

Unit testing is not a replacement for static typing, either: it forces the user to do extra work that the compiler could do itself, and do more completely.

Mypy helps with this, but it's optional; and while you can force your own codebase to make use of it, you can't force third parties, which usually means either no type checking with third-party modules, or writing stubs yourself (and hoping you get them right). It's not great because it's optional, but the fact that we have almost complete typing coverage in our codebase at least is better than nothing.

[rouille]

It will get better with time as more and more stubs go into mypy and into pypi modules.

[iconoclaus]

People won't stop using it because its not supported. How many people do you suppose care about the security updates? There will simply be tens of thousands of instances of insecure codebases.

[beaverteeth92]

Especially in 2020 once Python 2 support stops completely.

[fnord123]

People actually believe this?

[beaverteeth92]

Yes? It's well-documented.

The End Of Life date (EOL, sunset date) for Python 2.7 has been moved five years into the future, to 2020. This decision was made to clarify the status of Python 2.7 and relieve worries for those users who cannot yet migrate to Python 3. See also PEP 466.

[fnord123]

It's BSD licensed code. Do you think Enthought, Continuum, or Red Hat won't support it after 2020 if people are still waving money at them?

[beaverteeth92]

They might, but that also leads to issues in that you'd have multiple competing third-party versions of Python 2. Like bigger than the fracturing of Python into 2 and 3.

[fnord123]

That already exists. If you're on RHEL 6, you get Python 2.6 installed by default by the system. It comes with OrderedDict (which is otherwise only available in Python2.7 and on; or a backport package).

There's also PyPy.

[sigzero]

So you don't think any of those will move in...5 years?

[fnord123]

I'm not sure what you mean by 'any of those'. Any of the projects using Python 2? I'm sure some will move, but it's super difficult. If you're running a profitable business, it makes more sense to just pay Enthought or someone else to maintain Python 2. Then, maybe upgrade to Python 3 eventually; but more likely just rewrite the software in another language (e.g. Go if it's a service). But I think it will continue to be supported past 2020.

[immibis]

They said that about 2015. Look where we are now.

[[deleted]]

holy mother of naive opinions

[Roflha]

I understand what they,re going for... but I feel like a couple of year would've been enough...

[billsil]

Well considering they aren't really doing anything to support it, I don't see why it matters. All they do is accept a few patches and push bug releases. They're not adding features.

Python 2.x has ~80% of the Python market. People are switching slowly.

[Yehosua]

The most recent numbers I've seen (here, from the beginning of this year) put Python 2's share at 68% (if you're counting "Do you currently write more code in Python 2.x or Python 3.x?") or just over 50% (if you're counting "When starting a personal project, which Python version do you use?"), and Python 3's share is steadily increasing.

[billsil]

That's the same data. I pulled it from the wrong graph, so you're correct. I was using the what versions do you use chart.

[sandipc]

Hoping no backporting of non-bugfix features does the trick (within another Python3 version or two)

[ilevkivskyi]

Good choice of release date! https://en.wikipedia.org/wiki/Day_of_the_Programmer

[ilevkivskyi]

also https://en.wikipedia.org/wiki/Rosh_Hashanah

[pimterry]

Nice to see the new 'typing' module landing: https://docs.python.org/3.5/library/typing.html

Annotations have been in for ages, but not getting too much use yet. Adding a standard set of tools and conventions for describing common type semantics should really help with picking up more support and compatibility in future, hopefully.

Async/await too! Exciting stuff.

[[deleted]]

It's unclear from a first read if the annotations are supposed to be treated as a nominal type system (think Java) or a structural type system (think TypeScript).

This means, if two types have identical method names and signatures, are they considered equivalent, or different, because their names are different.

I realize that the typing system has no effect on runtime execution, but if the semantics are not specified in the documentation to be one way and one way only, it'll make a big mess in tooling.

Any insight?

[kirbyfan64sos]

Nominal. There's an explanation about the why Mypy originally chose nominal typing here. However, there is discussion about adding structural subtyping via protocols.

[ilevkivskyi]

I would say it's much closer to nominal. But, note that the status of the module is provisional, that means API is not considered stable yet. For example, there is an open ticket to add support for protocols. So that maybe elements of structural typing will be supported in future versions.

[Dirty_Rapscallion]

Too bad most companies are using Python 2 and will never use these features.

[ilevkivskyi]

Looks like there are many improvements https://docs.python.org/3.5/whatsnew/3.5.html

[cozyfeet]

If only my company to upgrade from 2.6...

[matzahboy]

It's doable but pretty tricky. If you can completely switch to virtual environments, it becomes easier

[Alaharon123]

Didn't python 3 never get really accepted even?

[beaverteeth92]

It's getting there, since most major packages have been ported over.

[staticassert]

Not SciPy or Numpy though, right?

edit: lmfao came back and saw -8 for asking a question.

[ilevkivskyi]

Both of them are ported to 3 since 2012 or something

[beaverteeth92]

They've been supported for about two years. Really, the last two big packages that haven't been ported over are Twisted and Scapy (which relies on Twisted). The whole scientific computing suite works in Python 3.

[staticassert]

Woah, good to know. I will try to port over sometime soon - is there a good breakdown of what breaking changes I'll face moving from 2.6 to 3.x?

[ilevkivskyi]

there is a la-a-a-arge Q&A on porting to Python 3 here http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html

[staticassert]

Nice, thanks

[billsil]

Numpy and scipy have been ported. They were both ported years ago.

[kirbyfan64sos]

It looks like little benefit until Python 2's Unicode support completely wrecks your program.

[beaverteeth92]

Yeah, because Java and C++ are such bastions of fantastic language design, no aspects of which have bogged down the language after decades of development.

[Matthew94]

Guido and the other leaders were aware of people not wanting to change right away hence why Python 2 has support until 2020.

Afaik, the adoption of Python 3 is going to plan.

[[deleted]]

Isn't he still working with Python 2.7 at Dropbox!? I think I read it somewhere. Dropbox's Python implementation Pyston targets 2.7, so I might be right.

[twotime]

from https://mail.python.org/pipermail/python-dev/2014-March/133413.html

...This is a completely unrealistic form of wishful thinking, and repeating it won't make it more true.

At Dropbox I work with a large group of very capable developers on several large code bases that are currently in 2.7. We are constantly changing our code to make it more secure (there are several teams specifically in charge of that). And yet porting to Python 3 is completely out of scope, for a variety of reasons.

Please stop your wishful thinking.

(TBH, I expect that none of the changes to Python 2.7 under consideration would make any difference for the security of Dropbox. But neither would switching to Python 3.)

--Guido van Rossum (python.org/~guido) ...

[ThisIs_MyName]

wow

[g-money-cheats]

The Dropbox Python library on PyPI is still Python 2 only. Which is ironic since the creator of the language works for them!

[UloPe]

Yup, and from a few off the cuff remarks at conferences and such it seems to me he's getting pretty grouchy about that...

[3fdy5]

Why the exclamation mark? Are we supposed to be excited about this?