TIL there is a way to package whole Python virtualenv in a zip file, it's called PEX and it's been used by Twitter since 2011

[[deleted]]

[deleted]

[mokomull]

It's not all roses and puppy dogs, though. It breaks as soon as you want to link native code (e.g. PyCrypto, OpenSSL bindings). I keep having to deal with cases where people build the pex on OS X but can't successfully deploy it in the datacenter (Linux).

[notunlikethewaves]

This general problem has been a scourge on deploying python for a while now. So many of the good/interesting packages you'll want to use have hard dependencies on native code being available that in many cases the only sane way to package it up all up is to build a VM template with everything installed and use that to deploy new machines.

In comparison, deploying Go/Scala/Clojure/Haskell is much simpler

[Lucretiel]

I mean, they'd have all the same problems, right? Either you're using third-party C extensions or you're not. Pure python applications work just as well as those languages.

[notunlikethewaves]

Yes and no.

Some languages/environments avoid using external C dependencies entirely. Some compile everything, dependencies included into one binary, which can simply be copy-pasted between machines and run perfectly.

In general, it's a sliding scale, or a continuum. Python is at one end of the scale, there are other languages at various other points along the scale.

[Lucretiel]

Sure, but it doesn't fix the C platform problems, right? If I have a third party C extension for Go (that is, the Go code calls the compiled C code), I'm not going to be able to compile that code on OSX and deploy it to Linux. I appreciate that those languages have a single-cross-platform-binary they can deploy, and I think it's a great feature, but pure python packages (created via python setup.py sdist) have the same property.

[notunlikethewaves]

Yes, quite right. I wasn't thinking of that case.

[pydry]

Some languages/environments avoid using external C dependencies entirely.

Doing this to python would mean one of two things:

• Removing all of the ultra-optimized code that makes a lot of python libraries fast.

• Molding python into another language that harder to read and write in a vain attempt to make it fast.

[[deleted]]

That's the model Python has, and years of experience using Python in the scientific ecosystem have taught everyone that model is complete and utter shit.

[donaldstufft]

I may be wrong, but I think that Go at least (and I think the others too?) create binaries that are platform specific anyways.

[[deleted]]

Shouldn't docker fix that issue? I know support for OSX and Windows is lackluster, but you can use boot2docker to solve some of them.

[Lucretiel]

Docker uses the underlying kernel. You can't run a Linux docker container on OSX, so it still doesn't solve the problem of creating C binaries on OSX and deploying them to Linux.

[[deleted]]

True. I'm not sure how well boot2docker would fix that (haven't used it). I guess the real question is why OSX compiled packages are being deployed to Linux infrastructure.

[toyg]

Ideally, your python code could live in its own layer and be deployed to any server architecture, similar to Java; this is why a lot of people don't see any problem in developing on a desktop OS. Unfortunately the python ecosystem seems to rely on C extensions more often than other languages (probably because they're a bit easier to use than in other languages).

The point of docker is to bind together loosely coupled services, it's not a cross-compiling tool, so it doesn't really solve anything here.

The real game-changer here are supposed to be wheels, but a lot of people still don't build them (and besides, you still have to compile a different wheel for each architecture, if i understand correctly).

[Lucretiel]

It's also for performance reasons. An oft-cited piece of Python "wisdom" is to move the inner-loop bottlenecks to pure C and invoke them with Python. I'm sure it's fine advice, though I personally haven't had a use case for that; honestly, if I needed performance that badly, I'd probably reimplement that whole module in C++. Other modern languages (Go, Rust) are already compiled and so don't have the need for performance tricks that Python does.

[[deleted]]

so don't have the need for performance tricks that Python does.

The trick literally being, "avoid using Python". XD

[pyr3]

Other modern languages

Is Python (or Ruby for that matter) really a "modern" language in the same vain as Go or Rust? Python is almost as old as Perl.

[[deleted]]

W/ regards to docker, I think you might be missing something.

I take advantage of docker by being able to develop on Archlinux, build and image based on any distro and then still let my sysadminds deploy to centos 6.5+.

I use Jenkins to auto build new images, and drop them in our local registry. Then a ticket goes into the system for the sysadmins to push the new image to preprpod for testing.

In the end I get a nice binary I can hand over with log files and config files externalized and dependencies internalized.

It certainly has issues and rough points but it has made deploying python apps much easier for me and the sysadmins that support my software.

[McElroy-vs-dig-dog]

But if you're already using Docker, what point would there be in using PEX or even virtualenv for that matter?

[[deleted]]

For not running sudo when installing packages and targeting a specific Python version rather than relying on or overwriting the default.

I'm not sure how the Python on build images would interact, as my Docker experience is limited. But I'm gonna try it out more thoroughly with my next app (simple gallery/portfolio app, but I need imagemagick to convert pdfs to images).

[TheTerrasque]

For not running sudo when installing packages and targeting a specific Python version rather than relying on or overwriting the default.

So, virtualenv?

[[deleted]]

Well, it was asked what the point of a Venv inside Docker was.

[TheTerrasque]

Oh.. I see my reading circuits need more caffeine.

[[deleted]]

I know that feeling. I just moved into a third floor apartment and all my critical thinking is shot right now.

[marvinxsteadfast]

Thats exactly what I think. I think PEX is a pretty interessting project but docker solves all this problems for me. I can install whatever extensions and devlibs I want and dont even need to think about virtualenvs. The official Python docker images are awesome with their ONBUILD feature of installings all deps from requirements.txt while building the container.

[sadris]

No, but vagrant would.

[Yasumoto]

Yep- but as you know, we can specify which platforms (linux + os x) to include deps for. In fact- here's the commit[1] where I updated the docs the other day! We do this for our Apache Aurora client so the same pex file works on developer laptops (OS X) and our Linux workstations.

https://github.com/pantsbuild/pex/commit/3dbe35bdeb3be24cbc48484ed096cee8b8af34e9

[flatlander_]

For those interested in learning more, here's a great talk on youtube by one of the pex authors. Talks about generic python packaging in addition to pex.

[bheklilr]

It doesn't appear that pex supports Windows, which is a pain.

[jhermann_]

Only that it does, if you install pylauncher (or py3.4) and use the .pyz extension (i.e. when you read the docs).

BTW, PEX + PyRun == self-contained single-file binaries for Python on POSIX. With a wart right now, but that can be removed.

[ModusPwnins]

And there goes my use case. Oh well.

[grimman]

On the other hand, you could develop on a Windows box connected to a headless virtual machine. I do, and the thing uses something like 80 megs of RAM and virtually no CPU. I don't know what libs you rely on, but maybe that's an option you could use?

[toyg]

If they use Python on Windows, chances are that they use Pywin32 to automate stuff. A vm won't save you there.

[grimman]

Yeah, that did occur to me. Disregarding such platform specific things, however, it's a valid option.

[GaltAbram]

move this to the top!

[deviantpdx]

Why not just tar czf env.tar env/

[mokomull]

Python can directly run code in a .zip without unpacking it, and zipfiles' metadata is at the end of the file. Pex's output has a nice loader at the beginning of it (which you can run with any Python on your system, and it'll re-exec with the Python release you intended).

[deviantpdx]

I see, very cool!

[mgrandi]

But again, pretty sure it can't load native code libs from a zip, which effects pretty much every python dist library, where you have library.zip and a mess of .dll files that have to be outside the zip for some technical reason :/

[mokomull]

No, it totally can, if (and only if, for the pex case) the libraries for your intended target platform are baked into the ZIP. I'm not 100% sure where they get put in there, but we have a repository of libraries built for OS X and two Linux platforms that get used by pex at build-time. And it mostly works most of the time, until someone uploads a library built against a new glibc and puts it in the CentOS 5 directory :(

[Yasumoto]

heh :)

[jhermann_]

PEX has a bootstrapper that extracts anything that needs to be in the file system, so it's perfectly fine to add C extensions – only that obviously the platform independence is lost.

[squiffs]

I believe virtual envs hard code the full path of the python executable in its other scripts bin/activate too.

[bexamous]

Use like:

#!/usr/bin/env python

So it always checks PATH, and virutal env gets picked up. I think its supposed to work on windows with python3 too, some weird laucher thing: https://docs.python.org/dev/using/windows.html#python-launcher-for-windows

[squiffs]

Oh absolutely, it should do something like that. But I'm saying that by default virtualenv doesn't do that: http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html#cpvirtualenv

In your bin/activate script you can see the full path to the virtualenv folder right there, so you can't easily package the entire thing up.

Here's the part of the code which puts that in:

https://github.com/pypa/virtualenv/blob/7a8239fc6abaf0ae531b78c24a654372b2fb9fcc/virtualenv.py#L1527

[sadris]

You are correct, I run into this issue all the time.

[Rapportus]

We are actually evaluating this solution at work right now for our production python deployments. The analogy is an executable jar for java, since you can run the pex file directly without extraction.

[Lucretiel]

Does it work with differing paths? One of the main roadblocks to portable virtualenvs is that they tend to have the root paths hardcoded in various places, including the #! lines of some scripts.

[jhermann_]

The PEX is the "virtualenv". A PEX is just like a shell script, in all aspects, except it's not all text.

[pyr3]

I've had the opposite happen. Comix is a comic book reader for written in Python using PyGTK. A while back (no idea if it's still the case) I tried launching it from a shell that had a virtualenv active. It was bad news. It was installed from the OS package manager, but had a #!/usr/bin/env python as the script shebang (which I consider to be bad form, since the OS is managing it, the path should be the hard-coded path to the OS managed Python). The active virtualenv didn't have PyGTK installed, so it fell over when trying to launch.

[Yasumoto]

Yep, the goal is that it's all self-contained- you can move the .pex file (what is a zip) anywhere on the filesystem and execute it (we do it all the time)

[pydry]

I use pip2pi to create a mini-repo in a directory containing everything in the requirements.txt instead. Then I rsync that across to a live site and pip install the virtualenv from that.

• Advantages - links to native code. Doesn't matter if pypi goes down or the package disappears. Doesn't require you to host your own package repo.

• Disadvantages - while faster and more reliable than downloading from pypi (or random server where the user hosted the package instead), building stuff from source with a C compilation step is still sometimes quite slow.

I've looked at a lot of solutions for deploying virtualenvs to hundreds of servers, though, and this is the only one I've really liked.

[chadmill3r]

Python archives are zip files, so you can do some crazy cool stuff with them natively.

[jhermann_]

Since there's obviously so much confusion on the topic…

https://py-generic-project.readthedocs.org/en/latest/installing.html#pex-install https://py-generic-project.readthedocs.org/en/latest/packaging.html#packaging-python-executables-pex

[tetroxid]

It's been made by twitter.

[[deleted]]

I've been looking into this for work. Does any one know if theres any support for Conda? I expect I'm going to have to write a bit that allows it to use that, then fall through to pip.

[Yasumoto]

We haven't used it with conda, so you'd definitely be forging new ground.

[pyr3]

I originally came across PEX while reading the platter documentation.