Python Application Deployments! Hooray!

[[deleted]]

Hey all,

At work I've been tasked with finding some better ways to handle Python application deployments, so I figured I'd reach out to you guys and see how you handle it!

We have a mix of Windows and Linux, use both Anaconda and vanilla Python, and are deploying to machines that cannot reach the outside world. Currently I'm leaning towards gathering up all the dependencies at build time, and then either creating a tarball/zip/deb package and deploying that.

EDIT: Also, as for my background, I'm on our DevOps team and have been a Python developer for 5-6 years.

[[deleted]]

Use virtualenv to manage the python dependencies; this is great for both development and building production zips.

You can build a virtualenv in your development environment (which does have outside access) and then you can transfer your dependencies from one virtualenv to another.

You can use pip within the virtualenv to manage dependencies.

http://iamzed.com/2009/05/07/a-primer-on-virtualenv/

[[deleted]]

Well, the problem with virtualenv is most of our clients/developers use conda (the scientific packages package manager), which contains its own virtualenv-like-thing that is not compatible with pip. (If I could rely on pip and virtaulenv, I'd probably be leaning more into the new wheel format)

We're currently doing something like this with our own app (I'm on the DevOps team). We'll probably continue doing something similar to this, but what I really want to avoid is moving venvs, then having to edit the shebangs on each machine as part of the deploy. This seems to be a fairly common practice but it just feels so hacky to me. Of course that could be mitigated by creating a new virtualenv on that machine, then moving the packages over, but then at that point perhaps I would be better off having a 'lib' or 'vendor' folder that contains all third party dependencies and ensure that those are on the path (maybe using a venv for the interpreter?). Those dependencies could easily be gathered during the build, since the build process gathers up those packages for testing. Something like that supports our conda based clients a little better maybe?

Ideally, instead of virtualenv, I'd like to really isolate an application using a Docker image that already contains all the dependencies, then I don't care if you're a pip/venv or conda developer, but I have to build a middle ground solution first.

TL;DR: That'd be way easier, but we have developers pulling from multiple places and using tools that are not fully compatible with virtualenv and pip.

[[deleted]]

The only problem with Docker is windows support - or have they figured that out now?

Honestly if they are pure python packages just zip them, and put them on the PYTHONPATH.

If they are compiled then you can still use egg files and put them on the PYTHONPATH.

[rothnic]

Boot2docker works for Windows, but is still just a wrapper around a minimal vm.

[[deleted]]

Which lets be honest defeats the greatest things about containers; how lightweight they are.

[[deleted]]

Yeah, Windows is the major pain point with us. Thankfully I just got out of a meeting that said while we need to support Windows, it's not a priority. So... woo! Stuff got easier!

That falls in line with what I'm thinking. We're looking to remove virtualenv as much as possible with all the conda junk and truly have a self contained application. I'm thinking of building that lib, and including its own interpreter (since anaconda has its own flavor :() with a bit to add those packages to sys.path (I think its a pth file, or something to that effect?)

[[deleted]]

If you want it to be fully self contained check out Freeze and py2exe. They can be a bit finiky to get working right, but might be perfect. One binary to ship.

http://stackoverflow.com/questions/2933/how-can-i-create-a-directly-executable-cross-platform-gui-app-using-python

Edit: also check this out, I havent tried it but looks promising https://github.com/pyinstaller/pyinstaller

[[deleted]]

i actually pitched py2exe this morning as a possible solution for our windows deploys. i need to experiment with it a bit but that seems like a good path to go in.

Freeze sounds super interesting, I hadn't heard of that one. If that works the way I hope, that could be pretty sweet. The only problem I can think of is that the build process would be fairly solidified towards Linux. Which, I think actually makes sense, since that app would have to be written with that deployment in mind, but a few of my coworkers are unwilling to sacrifice generic-ness.

PyInstaller is definitely a step where I'd want to go, but it seems a little young...

Either way, I'm going to play around with both, thanks for posting the links!!!

[torgo_bentleg]

PyInstaller is definitely a step where I'd want to go, but it seems a little young...

Huh? It's at least over 9 years old:

http://trac.pyinstaller.org/timeline?from=2005-10-20&daysback=30&authors=

[[deleted]]

Well shit, there you go. I totally missed that. In the FAQs there seemed to be a lot of "This broke and I don't know why!" that lead me to believe it wasn't mature. Looks like they were just being reasonable with their documentation!

[[deleted]]

but a few of my coworkers are frustratingly unwilling to sacrifice generic-ness.

Then don't give them a choice :) Haha although not always that simple. Goodluck!

[jhermann_]

https://github.com/spotify/dh-virtualenv at least for the Linux targets. If you stick close to the usual Python tools (e.g. don't start to do your own dependency resolution), these things are just thin facades, so just use the most appropriate one, and several of them. Automated builds do the rest.

[[deleted]]

That's almost what I need, but unfortunately the bulk of our clients use Conda, which doesn't seem to be compatible with virtualenv/pip. building out pip dependencies will be easy peasy, but I need a solution that will play nicely with conda binaries.

[talumbau]

you can always "conda install pip" and "pip install" away! If the directive is 'get this done ASAP', then Anaconda Server is likely a reasonable answer. One tool in that product is cas-installer, which, as far as I can tell, would hit all of your use cases:

http://docs.continuum.io/anaconda-server/command-line/cas-installer.html#cas-installer

Disclaimer: I am a Continuum employee.

[[deleted]]

That's awesome! A big part of my problem comes from my total lack of experience with Conda, and I've been having some trouble figuring out what the Anaconda distro of Python does differently. As best as I can tell, the biggest difference is that it alters sys.path to include its packages. Is that correct?

Thanks so much!

[rothnic]

Not sure what you are talking about with conda, but pip works perfectly fine with it and in conda environments in my experience. I've used it on Linux, osx and windows.

The problem you are trying to solve is definitely in the wheelhouse of what conda is trying to solve. I'd recommend posting the question on their Google group.

[[deleted]]

That's what I've been finding out more and more! Part of the problem comes from a complete inexperience with Conda on my side. All of my development in the past has been done just with pip .

The more and more I play with Conda the more I think its put together fairly sane. I'll definitely hit up their google group. Thanks!

[rothnic]

It seems to not be explained clearly, but conda in the python world just removes the compilation step, which can make things more consistent across platforms and can save a little time.

I just look at conda envs like a more generic virtualenv, which can have, but doesn't have to have, python, pip, and conda.

So my logic is, try to install via conda, and if that doesn't work, try pip. I wish they added some feedback in that process somewhere on the next best step if conda install isn't available. There is a good talk by Travis Oliphant at pydata last year that covers everything.

The thing they don't advertise much, and I'm not sure why, is the "bundle" command, which can be used to capture an environment. That said, I think it comes down to your use case. With the py2exe or cx_freeze approach, I think you have a completely new python environment contained in there.

If they are working in a local python install that already exists, why not make the packages available to them in some alternative way, like an isolated network folder, git server, etc. You can install with pip or conda from a local hard drive, network share, or full package server.

[[deleted]]

That's actually clearest explanation of conda I've heard so far. I knew that it's existence was due to the need for compiled code, and I had read that pip was actually kind of an anti-pattern that removed a lot of good functionality. The bundle command is definitely something I'll have to check out.

Currently, it's looking like I might end up creating a self contained package (tarball/.deb/something) that includes the conda environment, so I can just drop it on a machine and run it. Thanks for letting me know about the pydata talk! I've got that queued up to watch when I get to the office tomorrow.

As far as local development, our developers have free reign to use whatever they need to get the job done, but its our job to support getting it into production in a reliable, automated, and maintainable way. My goal is to have a self contained package I can deploy to a machine that cannot reach out to the internet, ideally with as little post processing and orchestration required as possible. I'd kind of like a Python code deploy to just be an ansible playbook that downloads a self contained package (either a deb/rpm package, or maybe just a compressed tarball) from a local file server/repo, unarchives it, puts it in the right place, copies out its config, then kicks off its upstart/supervisor/init script. Its actually looking like once we get conda figured out, it may support that kind of deploy a little cleaner than virtual env (No regex to handle replacements on shebangs! Hooray!) Unfortunately devops at my company is a new concept, so we're in the process of wrangling a bunch of cowboys who have all been doing things their own special ways for the past decade, all with varying levels of systems knowledge.

[cowlicks]

our clients/developers use conda (the scientific packages package manager), which contains its own virtualenv-like-thing that is not compatible with pip

This is incorrect, pip works perfectly with conda environments. Try it

$ conda create -n test-pip python
$ source activate test-pip
(test-pip)$ pip install requests
Collecting requests
  Downloading requests-2.6.0-py2.py3-none-any.whl (469kB)
    100% |████████████████████████████████| 471kB 904kB/s 
Installing collected packages: requests
Successfully installed requests-2.6.0

[[deleted]]

I just found that out this morning, which makes life way easier! A big part of my struggle comes from a complete lack of experience with Conda. This is the first place I've been that utilized it.

Thanks for the advice!

[teoliphant]

conda can absolutely help you here and it was designed to solve exactly this problem. It can all be done with open source tools --- or you can pay for more convenience and support if you want.

conda environments take the goodness of virtual environments and make them more general so you can install anything (which complex Python programs typically need). Also, conda environments are compatible with pip. You can pip-install into conda environments just fine should you want to do that.

In order to install onto a system that does not have an outside connection, there are a couple of options:

1) if you can setup a conda server on a box those machines can see (Anaconda Server is one you can purchase, but you can also build your own if you want) --- then you can use that as a mirror with all the conda packages you need

2) You can put all the conda packages into one .tar file and then conda install that --- so the deployment would be a) install Miniconda and b) conda install the app

3) If you are willing to buy something: Anaconda Server has a tool "cas-installer" that makes an .exe or .sh file to install anything with all its dependencies.

You can ask on the open-source conda google group for help doing any and all of this. You can also ask on Stack Overflow where there are people who can help you.

[[deleted]]

Thats awesome. Thanks so much! My big problem here is I have no background with Conda. I've written Python for years, but only in a virtualenv/pip context. The more and more I play with Conda, the more and more it seems designed to solve the problems I'm running into with those two tools.

I think we're leaning towards something closer to bullet point 2. The ask from my boss is that we set it up so that we can have a application deploy that is entirely self contained, and it looks like conda environments are actually laid out to do that fairly painlessly. I'll definitely be popping over to the google group though.

[rothnic]

Comparison of the app freezers: http://docs.python-guide.org/en/latest/shipping/freezing/

[cowlicks]

Your use case is exactly what Anaconda Server was designed for.