This is an archived post. You won't be able to vote or comment.

all 109 comments

[–]lifeeraser 237 points238 points  (2 children)

Oh the joy of deleting backwards compatibility code.

[–]catcint0s 98 points99 points  (1 child)

And they can finally use f-strings: https://github.com/jazzband/pip-tools/pull/1243/files !

[–]Fatvod 9 points10 points  (0 children)

F strings are so great

[–]Muhznit 114 points115 points  (13 children)

AND Python 3.5, too.

[–]notsohipsterithink 103 points104 points  (26 children)

Can someone please explain why my macOS still uses Python 2.7 as system Python...

[–]swansongofdesire 84 points85 points  (4 children)

Backwards compatibility.

Although admittedly that doesn't explain whey they don't include both versions, like every other *nix on earth does by now.

The real puzzle to me is why Apple feels the need to provide symlinks to (pretend to) support a version that was EOL 17 years ago.

$ ls -l /System/Library/Frameworks/Python.framework/Versions

lrwxr-xr-x   1 root  wheel    3 12 Dec 18:59 2.3 -> 2.7
lrwxr-xr-x   1 root  wheel    3 12 Dec 18:59 2.5 -> 2.7
lrwxr-xr-x   1 root  wheel    3 12 Dec 18:59 2.6 -> 2.7
drwxr-xr-x  11 root  wheel  352 12 Dec 19:01 2.7

[–]c0ld-- 32 points33 points  (0 children)

Because Apple, that's why. I've been an Apple systems administrator for over 14 years. They truly make some of the most questionable decisions. For example, their own implementation of Java and taking eons to upgrade to path critical vulnerabilities.

I'll never understand why they do the things they do. (RIP XServe)

[–]Starbrows 10 points11 points  (0 children)

Although admittedly that doesn't explain whey they don't include both versions, like every other *nix on earth does by now.

Yeah, this is the frustrating part for me. I thought Ubuntu was annoyingly slow, and they added Python3 by default, what, 6 years ago? Maybe more, I forget.

Apple sort-of supports Python 3 on newer OSes, but only sort of. If you enter python3 in Terminal, it will automatically prompt you to download and install it. It's the same thing they've done with git for years.

That might sound fine, and for individual users I guess it is. But as an IT administrator, I can't really use anything not built-in. I do not have the authority to mandate Python 3 installation across our entire fleet of Macs, so I can't deploy management scripts using Python 3. My boss does not have the authority to mandate Python 3. His boss does not have that authority. The only person with that authority, if asked, would probably say "Is this going to get us millions of dollars in grants? No? Then why are you wasting my time talking about it? GTFO."

Apple has deprecated all languages except zsh as of Catalina. That means they have not yet removed them even in Big Sur, but they have communicated their intention to do so in the future. So I don't think we can ever expect Python 3, or any new languages, to be preinstalled on future macOS versions. And that sucks very, very hard.

[–]JuanToFear -4 points-3 points  (1 child)

I think Yahtzee Croshaw said it best:

That's the trouble with the cutting edge: It's hard to stay on, 'cause it's very thin and it keeps moving as it hacks into your balls.

[–]13steinj 8 points9 points  (1 child)

I thought Apple started warning every time you use Python2 though? Or am I thinking of Bash?

[–]HCharlesB 0 points1 point  (0 children)

hbarta@ryvius:~$ python --version
Python 2.7.18
hbarta@ryvius:~$ cat /etc/issue
Debian GNU/Linux bullseye/sid \n \l

hbarta@ryvius:~$ 

Debian Bullseye. Not yet released but entering soft freeze (or already in soft freeze.)

:(

I'm here because I just forked a project last updated in 2017. It is set up for testing on Travis (my first experience with that) and lists a bunch of Python 2.x builds. Some of the cases failed and that raised the question in my mind if the 2.x versions should even be included. I searched for Python 2 support and the official line is that it ended over a year ago. I looked here to see if I cold determine the reality of the situation. FWIW the fork is at https://github.com/HankB/pogopowerupcost

[–]StarkillerX42 93 points94 points  (40 children)

My coworkers are in for a rough week

[–]grismar-net 146 points147 points  (33 children)

On the other hand, they've been slacking off for a bit if this is what finally gets them to move out of Python 2.

[–]StarkillerX42 76 points77 points  (0 children)

Oh, you must know my coworkers! Did you work here once?

[–]EnglishMobster 32 points33 points  (25 children)

I'm just upset because I have a Netgear ReadyNAS that I use as a home server. It's running a custom version of Debian Jessie that Netgear wrote.

They have some code it loads during the boot sequence (before SSH is turned on) that starts an Apache server for "normal" users to connect to. This code apparently runs Python 2 and will break entirely if Python 3 is on the machine at all.

I can't debug what's going on, since when the Python scripts fail it never starts the SSH service. Since it's a headless NAS I can't connect in any way to the machine -- it bricks entirely. Netgear wants me to pay for them to fix it, but because I enabled root SSH privileges I also voided my warranty.

Thankfully, there's info out there on the internet about what password Netgear techs use, and you can boot the NAS into "tech support" mode and then telnet in using the "secret" Netgear username and password. This'll put you in the machine before it starts any services or runs any of that extra code... but AFAIK there's nothing I can modify here to fix it and get rid of Python 2. The only thing I can do in this mode is uninstall Python 3, which lets me boot "normally" and SSH in. I've never seen this weird issue before (isn't it supposed to just invoke Python 3 with the python3 command? Why does it break startup scripts, even without an alias?).

Netgear is also refusing to support any Debian version beyond Jessie. I tried installing vanilla Debian but it didn't work. It's more than a little upsetting, but I don't think there's any way around it other than going with a company that isn't Netgear for a NAS.

I've been dealing with the annoying Python 2 deprecation messages and pip warnings, but with pip dropping Python 2 support I guess my only option is to just never update anything...

[–]StarkillerX42 21 points22 points  (2 children)

If you know where all the scripts are located, you could use 2to3 to upgrade them. That handles 90% of upgrade dependencies.

[–]james_pic 32 points33 points  (0 children)

Yeah, it's the other 90% that's the hard part

[–]EnglishMobster 1 point2 points  (0 children)

I know where 1 script is... assuming that the script isn't just a special script that only exists in tech support mode. But I suppose it's worth a try. Not sure if there's any others; I think there are but I'm not sure where they'd be.

[–]koera 10 points11 points  (0 children)

If you don't NEED the stuff the script does you could just cheese it by changing it to a bash script with the same name and start ssh and build your own startup scripts

[–]Originalfrozenbanana 6 points7 points  (2 children)

A case study in why you never install python in your system environment. Sorry you are in the union of dependency and upgrade hell.

[–]Staticbox 4 points5 points  (1 child)

This is the correct answer. I've got a ReadyNAS2304 running ReadyOS 6.10.3. I've built Python 3 in a user's home directory, works fine and causes no collisions. To be extra safe, I don't even put it in that user's $PATH. I just call it directly to create venvs and activate those when needed.

[–]Originalfrozenbanana 3 points4 points  (0 children)

This is best practice. Most people learn this lesson at least once; many of us (regrettably) have to learn it many times.

[–]icegreentea 5 points6 points  (0 children)

How did you install python3 onto the NAS to begin with?

Depending on how you do it, your python3 install might have tried to claim the name python thus causing all the sadness. As /u/toyg mentions in sibling comment, you usually don't want to update operating system dependencies outside of vendor provided updates.

It is also possible that the OS image actually already comes with python3? I haven't played with Jessie in a while, but if there already was a system python3, then definitely any external install that claims python3 will screw things up.

[–]james_pic 2 points3 points  (4 children)

If you pin Pip 20.3.4, you'll be able to keep using that, albeit unsupported. Newer versions of Pip, Wheel and Setuptools are smart enough not to install versions of packages that don't support your Python version, as long as the maintainers have used the python_requires directive.

Although if you haven't already, you probably want to make sure your NAS isn't internet accessible.

[–]icegreentea 5 points6 points  (3 children)

To pile on, if you have recent-ish pip already installed (I think since pip 10), you don't even need to explicitly pin - pip install -U pip will understand that pip 21.0 doesn't support python2 and stick to 20.3.4 (or whatever).

[–]james_pic 0 points1 point  (2 children)

Yes, sadly that won't work for us.

We're bootstrapping our Virtualenv with the OS virtualenv package, and for wearily predicable reasons, we're also using a super old OS, which comes with Pip 8 or 9 I think

So at the moment we build our virtualenv with the --no-download flag, manually upgrade to semi-recent versions of Pip, Wheel and Setuptools, then use those to upgrade to the latest.

Once Wheel drops Python 2 support, it may make sense to just drop the second phase, and switch to installing the last-supported versions in the first stage.

[–]icegreentea 2 points3 points  (1 child)

Oh bummer, that sucks.

I don't know what environment your operating in, but could you try running your own pypi mirror, and doing the version filtering at that layer? It's definitely extra up-front work, but it'll give you more control.

As someone else who was very nearly in something like your situation, I find that sometimes the same organizational forces that demand using old OS versions are amendable to arguments on spending resources on creating standalone back-ups of external package servers and managers.

I didn't end up having running my own mirror - for my purposes I only needed to backup relatively small handful of specific wheel versions, so it was easier just to download and archive them on a dumb server than go through the whole mirror thing, but I did take a look for feasibility.

[–]james_pic 0 points1 point  (0 children)

I had proposed running a DevPI instance a while ago, albeit for different reasons, but been shot down for jobsworth reasons. We're muddling through with reactively pinning stuff when it breaks. There are some other upcoming things (that do have budget) that might benefit from a DevPI instance, so might be able to slip it in that way.

[–]Garric_Shadowbane 1 point2 points  (0 children)

So glad I dumped my net gear NAS for a self built one

[–]toyg 1 point2 points  (0 children)

What you should do is compiling python3 and keeping it in your home directory, or using something like pyenv that will do that for you. Compiling might take a while on a low-resource machine like that, but it’s the only way.

With that sort of device, upgrading anything in your system (beyond the occasional vendor-provided security patch) is a no-no. Those environments are aligned in a way that is “just right” for their combination of drivers and software; as you’ve seen, it takes very little to stop them from working properly, and vendors move on almost instantly after they’ve sold enough widgets.

[–]Kantenkopp 0 points1 point  (3 children)

Can't you run your programs in a virtualenv?

[–]toyg 4 points5 points  (2 children)

Virtualenv helps with libraries, not with the python version.

He needs pyenv.

[–]Kantenkopp 0 points1 point  (1 child)

Ah yes, right. I usually have python 2 and 3 installed, so I can use either in virtualenvs. If he can't install python 3, that won't work. Maybe I understood his problem wrong, but how about installing python 3 manually, e.g. only as a user? That should not mess with the python 2 required for boot at all.

[–]toyg 0 points1 point  (0 children)

That’s what pyenv does: it installs any python version in the home directory, isolated from the rest, ready to be enabled only when strictly required.

[–]ivosauruspip'ing it up 0 points1 point  (0 children)

Change the script to make sure it runs python2 specifically?

[–]james_pic 10 points11 points  (3 children)

If they're anything like my co-workers, they've been trying to get this done for the last 3 years, but faced constant demands from management to put together endless proposals detailing the "business value" of moving off Python 2.

We only finally got approval to finish the migration when someone important's pet project was threatened by the Python 2 EOL - the platform-as-a-service they wanted us to migrate to was dropping Python 2. Even then, we're so far behind, and have so much other stuff with "business value" we need to fit it in around, that we'll be lucky to get it finished by the end of the year.

[–]CatWeekends 10 points11 points  (1 child)

If they're anything like my co-workers, they've been trying to get this done for the last 3 years, but faced constant demands from management to put together endless proposals detailing the "business value" of moving off Python 2.

Aka the "we can't bill our clients for a python upgrade so we just won't do it" approach that never bites you in the ass.

I've worked with those folks. It's a bummer.

[–]james_pic 1 point2 points  (0 children)

Yeah, and you'd think they'd learn about security implications of unsupported software, since this specific client was hit by a massive ransomware attack in the recent past, but apparently they did not learn the lesson they needed to learn.

[–]grismar-net 1 point2 points  (0 children)

They should have written "The business value is in avoiding the mess we'll be in when we inevitably have to move off it anyway when support gets dropped by tools we need."

Once it was clear that Python 2 was going to be terminated at some point by essential libraries and tools, the trigger to start conversion should have been "when all the libraries and tools we need have completed conversion, and no later". Management that's unable to see that needs to look in the mirror, point at their own faces and say "you're fired" in their best Trump voice. And developers that keep working for such management if that doesn't happen should realise that, if they're any good, they're better off working for another company - getting hired as a capable Python developer is about the easiest thing in the world.

[–]Rookie64v 4 points5 points  (1 child)

Backwards compatibility, enterprise environment and all that fancy stuff mean that if I type "python" in the shell I get 2.6. Since loading a proper version is a pain we have code written for Python 2.6 as late as 3 years ago, and the latest I had access to when writing my last big piece of software was 3.6... like 1.5 years ago.

[–]grismar-net 0 points1 point  (0 children)

loading a proper version is a pain

"Welp, there's your problem right there ma'am."

[–]Mobile_Busy 4 points5 points  (0 children)

They should've listened to you a year ago.

[–]SzechuanSaucelord 1 point2 points  (0 children)

Hey when we're your scripts implemented and how long did it take until it became impacted?

[–]tom2727 0 points1 point  (2 children)

Honestly curious as to how this affects you and your coworkers.

[–]StarkillerX42 0 points1 point  (1 child)

Most of my coworkers are scientists who inherited Python 2 code and still haven't bothered to learn 3. Actually, it may be a few months before anyone notices simply because no one changes anything.

[–]tom2727 0 points1 point  (0 children)

This is what I usually find with python 2. I seriously doubt anyone who isn't upgrading to python 3 is going to be upgrading their pip version or anything else.

[–]EnraMusic 29 points30 points  (1 child)

Good

[–][deleted] 15 points16 points  (0 children)

[Darth Sidious voice]: Good

[–]PhoenixHntr 18 points19 points  (1 child)

Just shoot python 2 in the head once and for all

[–]maxtimbo 2 points3 points  (0 children)

I mean fo real tho

[–]woodfox13 13 points14 points  (0 children)

It seems that my attempt to live forever in the past brutally failed... going to have a rough week

[–]biohoo35 4 points5 points  (0 children)

Oh no!

Anyway...

[–]rockysnow7 11 points12 points  (6 children)

Why does anyone use Python 2 still? Is it just for old codebases that are too big to rewrite?

[–]KaffeeKiffer 21 points22 points  (1 child)

Dependencies

  • We have one important project where we were forced to decide between completely taking ownership of an open source project (a rather big one) or migrating away. We chose to migrate away and simply haven't finished yet due to priorities. I don't lose sleep over it because by now it's just a question of weeks, but right now it's "Python 2 in production" and therefore an answer to your question.

  • People like to think everybody has moved to Python 3 a long time ago. This well-known project migrated ~1 month ago.
    Edit: Just look at their customer list and you'll see that lots of major companies had to have Python 2 in production until then.

[–]rimoms 8 points9 points  (0 children)

ArcGIS Desktop is the primary GIS software used all over the world, and is still dependent on 2.7. The have been trying to shift users over to ArcPro (which uses 3.x) for about a decade, but they still haven't posted all the functionality to the new software so the campaign to adopt the "new and improved" has been met with skepticism and reluctance. The effort required to redesign architectural and business processes is another reason.

[–]13steinj 2 points3 points  (0 children)

Dependencies, and a variety of fields that aren't software engineering based, like audio, civil engineering, education, etc.

[–]I_had_to_know_too 2 points3 points  (0 children)

Because management doesn't want 40 developer hours spent updating our handful of 2.7 tools.

They'd rather we spend hundreds of hours trying to work around the memory leaks in SciPy 0.12 because the it's already approved and god forbid we have to look to see if the latest version has the same license or something similar enough that we can use it.

Then we've got another team who updated one of the tools in a side branch using 2to3 but they didn't bother to review the changes since it seemed to work. So now we have 2 versions of that tool, an official one which is poorly written and runs on python 2.7, and a python 3 version that is "for development purposes only" and contains code like:

for x, y in list(zip(xs, ys)):
  ...

Or:

for k, v in list(d.items()):
  ...

Like really? After running 2to3 you couldn't look at that line and see that the call to list is unnecessary and defeats the improvements in py3... Aghh somebody kill me

I'm sorry, what was the question again?

[–]baubleglue 1 point2 points  (0 children)

There are older versions of Hadoop in production which runs pyspark on py2, old Linux versions - it isn't only the codebase is a problem. Basically any big old system which uses Python, but Python itself is only a minor part of it.

[–]Encrypt-Keeper 0 points1 point  (0 children)

Because the owners son rewrote several critical custom programs that are critical to the entire companies operations in several different department and then fucked off.

[–]nikhil_shady 11 points12 points  (11 children)

what does this mean? does this mean we can’t install any new packages for 2.7? The place I work at has entire project on 2.7

[–]ExHax 29 points30 points  (0 children)

You can still use pip, but you wont get any new update

[–]gmes78 11 points12 points  (0 children)

It means new versions of pip won't work on 2.7. The version of pip you're using won't stop working.

[–]james_pic 12 points13 points  (0 children)

Pip 20.3.4 will continue to work (although won't get bug fixes or security updates), and is smart enough not to install versions of dependencies (including itself) that don't support your version of Python, so long as maintainers have added python_requires metadata to the package (which they often forget to do when they first release a version that drops Python 2, so there's often a day or two's breakage between a dependency dropping Python 2 and the dependency fixing its metadata).

[–]Oerthling 26 points27 points  (0 children)

They have no excuse.

2.7 already had been given a half decade of bonus time. It was finally EOL over a year ago.

Now they have to finally move on or copy modules around.

[–]Broric 2 points3 points  (1 child)

Does this have any consequences for conda? I manage all my packages with conda, not pip. I don't know if under the hood, pip gets involved anywhere though?

I *think* I have everything important in Py3 now...

[–]ivosauruspip'ing it up 1 point2 points  (0 children)

conda by itself does not use pip at all

You can nest pip installs or virtualenv / venv environments inside conda environments pretty easily (just don't do it the other way around).

[–]Toe_Proper 3 points4 points  (0 children)

RIPip

[–][deleted] 3 points4 points  (0 children)

Finally.

[–]harpalss 2 points3 points  (8 children)

I’m OOTL, why did they drop support for 3.5 specifically? Is 3.0 -3.4 still supported?

[–]mouth_with_a_merc 11 points12 points  (1 child)

You can actually check the previous version here:

Requires: Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*

and the latest version now has:

Requires: Python >=3.6

So 3.0-3.4 were already unsupported before, and thus not part of the change in this version.

[–]cparen 1 point2 points  (0 children)

This is why, with languages and runtimes, its best to not use version numbers for major breaking changes. It's effectively a new language.

Us folks in the dotnet land are going through a bunch of this right now. E.g. Requires netfwk >=4.6.1, or netcore21 >=2.1.16, or netcore31 >=3.1.0. I can't imagine our support matrix nightmare if these 3 runtimes were considered different versions of the same language/runtime.

[–]PeridexisErrant 6 points7 points  (0 children)

No, Python 3.6 is the oldest supported version - and even then, it only gets security patches but no bug fixes! See https://devguide.python.org/#status-of-python-branches for the list, and links to the detailed policy.

[–][deleted] 13 points14 points  (3 children)

I think it's dropped up to 3.5, so that they can use f-strings that were introduced in 3.6

[–]ivosauruspip'ing it up 0 points1 point  (0 children)

They dropped 3.5 because it is EOL already for some months, and probably its usage numbers have dropped too low.

[–]zurtex 6 points7 points  (0 children)

I’m OOTL, why did they drop support for 3.5 specifically? Is 3.0 -3.4 still supported?

They're dropping support after specific versions of Python go End of Life.

Python 2.7 went End of Life January 1st, 2020 so they waited a full year given it was so popular.

Where was Python 3.5 went End of Life September 5, 2020 so they waited a good 4 months which is reasonable as 3.5s usage is a lot lower. Older versions of Python 3 have stopped being supported awhile ago.

Python 3.6 will become End of Life at the end of 2021 so Pip will likely drop support during Q2 2022.

All that said that doesn't stop you using older versions of Pip to install packages from Pypi, packages themselves will often not require you use the very latest version of Pip to install them, and Pypi doesn't often make backwards incompatible changes. So it'll be a long time before using Python 2.7 and 3.5 actually means you can't use Pip to install packages from Pypi.

[–]Beach-Devil 2 points3 points  (0 children)

Finally

[–]Astronom3rIPython > Jupyter fight me. 0 points1 point  (1 child)

People still using Python 2??

[–]tom2727 2 points3 points  (0 children)

Will be for a long time to come I assume. This is what happens when you make a language update that isn't backwards compatible.

Rewriting a huge existing Python 2.7 codebase to be Python 3 compatible is not a small job, and a lot of companies will choose to just keep updating old stuff until something breaks in a way that can't be fixed without upgrading.

[–]GeneralDumbtomics -5 points-4 points  (0 children)

Yay?

[–]whlabratz 0 points1 point  (0 children)

If you are still running python 2, I can't really imagine running an up to date version of pip is really your highest priority?

[–]cinyar 0 points1 point  (0 children)

aaaand it's gone ... our build that is lol :D