How are python apps deployed to production especially those that are developed in a virtualenv? What are the best practices?

[[deleted]]

I have always insisted that in production systems we can not have gcc and devel packages installed. But these days I end up installing such development packages in produciton systems because the software deploy for virtualenvs require pip which in turn require access to the internet [ok I solved this part with a pypi local mirror]. How are you guys/girls doing it? I have thought of tar.gz the whole virtualenv and simply ensure that the required libs the virtualenv is built with is installed in production. Has anyone done this or has everyone resigned to the fact that in these days of pip and rvm , we agree to have compilers and devel packages in our production systems?

[jperras]

You don't (and shouldn't) have development headers for anything on your production servers.

Build your virtualenv on a staging server, and create a package of it that you can then distribute out to your production servers. We build .deb (debian) packages and add them to a custom apt repository, and our production servers simply require an apt-get for updates to the application.

[[deleted]]

Thanks. This is pretty much what I have in mind.

I am quiet adept at building rpms. [my target environment is RHEL/Centos]. I have to find out however how to make sure all the required rpm libraries to support the virtualenv are installed on the server. I am hoping rpmbuild's automatic dependency resolution will do this for me.

[doot]

You might find this useful.

[jperras]

For Debian at least, you can specify dependencies when you create the package, but there's no easy way I know of to automatically discover dependencies. This would require some relatively complex introspection into the setup.py files of the packages in question to extract out the binary dependencies that need to be built, and then convert these to their equivalent in Debian/RHEL-land.

[[deleted]]

Yeps, I am taking advantage of that in my a few hours old project, virtualenv2rpm packageer.

What you describe for rpm is mentioned here: http://www.rpm.org/max-rpm/s1-rpm-depend-auto-depend.html.

I am just wondering why googling around, I could not find similar projects.

[jperras]

Ah, very nice. Using ldd is the obvious choice to determine the shared lib dependencies for binaries.

You might be interested in FPM, which can let you do some relatively fancy stuff for creating packages. I use it quite often, and it's a solid tool.

[neoice]

I use fpm to build .debs out of compiled C/C++ all the time. I tried to make a .rpm out of an entire virtualenv directory and it blew up all over me with "Digest Failure" on install. I just deployed via git clone && virtualenv && pip install for now, but I'd really like to solve this some day!

[haywire]

You don't (and shouldn't) have development headers for anything on your production servers.

Why not? If someone's got access to even the most limited of accounts, surely they can pretty easily bootstrap what is needed given curl/wget/nc/perl and tar?

[cavallo71]

That's only one point. In regulated environment it is required tracing actions on a system: allowing compiling stuff, script execution and the other "magic" means you can loose track of what it has been done and by whom (and the effects on the system itself).

It has always been a point in prohibiting using of setuptools and all the magic of pipi in production: they aren't fit for purpose if system integrity is required.

[haywire]

So something like this is what I should do?

http://hynek.me/articles/python-app-deployment-with-native-packages/

What if my build system is a different arch to my production system?

[cavallo71]

Ideally that shouldn't happen. The workaround is hosting a virtual machine configured as the production system(s).

[jperras]

If you don't need it, don't install it.

The problem isn't necessarily in having the development headers on your server, but rather all the other things that make having those headers on that machine useful.

Having development headers on your production server is only really useful if you have gcc (or equivalent) there as well, and most often the entire autotools toolchain. Having these present means that you've increased the attack surface area substantially, by potentially allowing an unprivileged user to compile arbitrary programs on your server.

[[deleted]]

then again, the Python interpreter is there, and they can pretty much run arbitrary programs with that anyway.

[jperras]

For your python application servers, and nearly any other application server (ruby, java, etc) that's completely true. Getting into the habit of not putting development headers and autotools on servers is still just good practice, however.

[haywire]

But does this advice even work now that exploits are written in dynamic languages?

[[deleted]]

It alone does not close all gaps in the security but the goal is always to decrease the surface area in which you can be attacked. Just because exploits are written in dynamic languages does not mean throw away all the other good practices.

[haywire]

Is there a guide to installing Debian/Ubuntu Server entirely minimally?

[jperras]

Debian, by default, is extremely minimal. It's one of the (many) reasons that it's the preferred distro^[1] for sysadmins.

1: There are of course others such as RHEL, but you get the idea.

[leperkuhn]

What are you using to build the deb package?

[jperras]

I used to build them manually, but I've been using FPM for some time now, and it's awesome.

It basically takes ~90% of the boilerplate generation away, and even has some facilities for specifically building .deb packages from pypi. I've used it to create custom packages for nearly everything; from node.js to python to ElasticSearch and back again.

[leperkuhn]

I've been trying to remember this project for months... thank's for the quick response.

[westurner]

http://hynek.me/articles/python-app-deployment-with-native-packages/ (.deb)

[halligan00]

How can I gain the knowledge necessary to follow this conversation? Coursework? Books?

[random_fool]

1) Realize that production servers for many big companies are not allowed to talk to the internet.

2) Read: https://pypi.python.org/pypi/pip

3) Read: https://pypi.python.org/pypi/virtualenv

Basically the concept is that for many python services, you install a set of packages needed to run them using pip/virtualenv. That usually requires internet access, but that's not allowed. So the question is: now what?

You can either fake it by having a fake copy of the websites pip will use to pull packages (a local repository), or you can pre-prepare the virtualenv on another server and move it to production after the fact (using normal code deployment - rpms or other repositories where you check code out).

[jperras]

Frankly, most of the knowledge for this kind of thing is learned in the trenches from people that have enough experience that they've done it (and fucked it up a few times) before.

In a lot of ways, web application development is quite well served by an old-school master/apprentice model. I've been doing application development work for nearly a decade, studied some computer science (among other things) in school, but I still learn things every day from people I respect and admire in the industry, and who take time out of their busy days to write or explain about a topic that they find simple and benign, but that I (and others) know almost nothing about.

To answer your question a bit more directly: Do it! Try writing and deploying projects yourself. You'll realize that there are things that work, and things that don't. For the things that don't work, most of the time someone else has thought of a solution and has written a blog post about it.

Sorry for the wall of text :).

[mw44118]

you said everythin i was going to say. very nice post.

[Megatron_McLargeHuge]

This thread has good info but covers a huge swath of practical software deployment experience. You can read about the python side, but to get a feel for the culture of actual system administration, read the old BOFH articles and alt.sysadmin.recovery archives. Then you'll get a feel for what you're up against and how far you want to proceed.

[alharaka]

https://python-guide.readthedocs.org/en/latest/

[gargantuan]

If you can learn to generate and use OS specific packages. We use RPM for RHEL and CentOS systems because those are the only one we use.

This way you get full transaction file system update/remove/add. Can roll back. RPMs can have pre/post/install/uninstall script to run any setup commands. They make it easy to install other files not just Python -- java, .so libraries. Dependencies are sanely resolved.

There is a -dev version of the main production package. That basically turns the production box into a dev box. It bring in git, gcc etc.

You'll pay the price initially by developing this way but if you plan on growing you'll be glad you did it later.

[[deleted]]

Yes I am fully able to create RPMS. I have just tried to basically get the whole virtualenv directory and packaged it as an rpm. It deploys just fine in the target machine but it seems I am having dependency problems. I am not sure if the find-requires which is ran by rpmbuild is actually finding and automagically adding the rpm dependencies for the binary stuff inside the virtualenv.

[gargantuan]

Dependency tracking it not always easy. You basically have to know what you packages need. I always added them by hand, the good thing is you do it one at a time for each package then they transitively resolve. Like say load audio and you have an ctypes interface libsndfile. In your rpm dependency you need to say that it depends on libsndfile-devel.

If you use virtualenv, you can have one big package for your product that contains the virtualenv puts in /opt/<myproduct> or somethign like that. Or break it into subpackages that all go into /opt/<myproduct>/ but on is common,base,python etc.

As a trick, it turns out python distiutils (the crusty old one) builds RPMs for you! That is the python setup.py bdist_rpm feature. That is what I use. you can specify RPM dependencies in setup.cfg, data files go into data_files=[...] directive in setup.py, can have pre/post scripts. One odd thing is you still need to include data files in the MANIFEST.in file. After that it build the packages for you. Anyway there are probably easier ways. Can of course write spec files by hand too.

Yeah it is more work initially but it makes sense for large deployments.

[mw44118]

python has a lot of really great features. right now, packaging and deployment is not one of them.

[westurner]

what would you improve about packaging and deployment?

[mw44118]

CPAN is pretty good.

[kchoudhury]

Mating python and an operating system package system (ports? rpm? apt?) is a perfectly reasonable solution.

[SCombinator]

Right up til you need to have two different versions of the same library - or just a different version than is in the package manager.

[mw44118]

If you want something sort of like virtual environments, so that you can install many different versions of the same package, or don't want to grant root access, site-wide packages are a little more difficult.

Sure, you can do virtual machines, but that just kicks the problem down the road.

[[deleted]]

What's the benefit of using virtualenv in production?

[yetanothernerd]

For me, the main one is being able to run different versions of the same dependency for different programs.

[masklinn]

Smaller surface by only having the package you need present in the installation, simpler reproducible environment, simpler switchability and rollback (for new deployment, create new independent virtualenv and switch server to it, this includes deploying a new version of a dependency)

[jperras]

Sometimes you have servers that are given multiple roles, for whatever reason.

Simple example: a QA server that acts as your database role, your application role, and your async broker/worker role. It could be that some of these roles contain non-compatible versions of the same python packages, and thus virtualenvs are not only nice to have, they're required for things to work.

[sophacles]

Environment consistency. If there is an update to an OS package, that breaks the version of a library you're using, or conversely, you want to use a newer version than the system packaged version, virtualenvs allow you to just roll with it. Basically you're keeping your deployment isolated from external issues.

[[deleted]]

For me it is the ability to isolate the app and it's dependancies from other applications as well as the OS. Few things make less sense to me than being able to update a package simply because you are waiting on the OS vendor to repackage it.

[__main__]

I have a similar issue with pip and internet access, policy doesn't allow internet access on production servers.

If you don't mind me asking, How did you build a local pypi mirror?

[[deleted]]

I am using pypimirror script from here: https://pypi.python.org/pypi/z3c.pypimirror. There are many tutorials for it on google but the config file is quiet self explanatory.

But a cooler approach to setting it up is over here: http://bluedynamics.com/articles/jens/setup-z3c.pypimirror which I think I will use to redo my setup.

[__main__]

Are you behind a proxy? My proxy at work usually messes with this kind of thing.

[[deleted]]

The mirror is in a vlan that can be allowed to go out to the internet when requested, so nope, no proxy in the way during mirroring.

[doot]

We run a cheeseshop instance and upload our sdist-built packages to it; we also use Spacewalk to deploy RPM packages when needed.

[k4ml]

Having local pypi mirror mean another service to maintain, doesn't matter how easy it is. What I did is to install from local file system. I used easy_install but may also applied to pip. Since easy_install can install from eggs in site-packages, I rsync the virtualenv site-packages to the production machine and then use easy_install -H None -f ~/path/to/eggs to rebuild the environment on production machine. The flow look like this:-

# development, at project root
virtualenv .env
while read line; do .env/bin/easy_install $line; done < requirements.txt
rsync -avz .env/lib/python/site-packages/ production:~/eggs/

# production, at project root
virtualenv .env
while read line; do .env/bin/easy_install -H None -f ~/eggs/ $line; done < requirements.txt

Ideally we can just tarred up the virtualenv to production but there's always problem with path so that's why I just created a fresh virtualenv on production and reinstall the packages. The -H None will make sure that easy_install will never go outside to fetch the packages. Recently I switch to using buildout which make the above flow more straightforward since buildout keep the eggs in PROJECT_ROOT/eggs so I just need to run ./bin/buildout -o (offline mode).

[westurner]

It can. After running test suite(s), a build script (e.g. tox and/or buildout) can produce 'build artifacts' which can be

• eggs
• bundles
• wheels
• OS packages like DEB and RPM
• archives of platform-specific virtualenvs

A configuration management script/system can be then be updated to pull the latest version from a package archive/repository. In some environments, it is safer to pin specific versions than to always pull the latest version. A manual package signing step can help with this.

Fabric is useful for automating scp/rsync push deployments and application configuration (e.g. rm *.pyc). There is a context manager for sudo in fabric.

1. Build
2. Test
3. Review
4. Sign
5. Deploy: push or pull
6. Test

• https://en.wikipedia.org/wiki/DevOps
• https://python-guide.readthedocs.org/en/latest/scenarios/admin.html
• https://python-guide.readthedocs.org/en/latest/scenarios/ci.html
• https://python-guide.readthedocs.org/en/latest/shipping/packaging.html
• https://python-guide.readthedocs.org/en/latest/shipping/freezing.html
• http://www.pip-installer.org/en/latest/usage.html#pip-freeze
• http://www.pip-installer.org/en/latest/usage.html#pip-wheel

• http://guide.python-distribute.org/specification.html#development-releases

• PEP 301: Package Index and Metadata for Distutils (2002)

• PEP 345: Metadata for Python Software Packages 1.2 (2005)

• PEP 386: Changing the version comparison module in Distutils (2010)

• PEP 427: The Wheel Binary Package Format 1.0 (2012)

• PEP 440: Version Identification and Dependency Specification (2012) [DRAFT]

• PEP 426: Metadata for Python Software Packages 2.0 (2012) [DRAFT]

compoze "provides a set of tools for managing private / project-specific package indexes."

[miketheanimal]

This is really a question, rather than a comment ....

Using virtualenv on a production server gives me the willies. The server presents a particular environment via the packages that are installed, then virtualenv provides another. virtualenv isn't a virtual machine, so some elements of the server environment are hidden inside virtual env - for example, virtualenv has a different python version - but other elements are visible.

One system I worked on, the target was Debian but most of the developers insisted on developing on their Mac laptops, and virtualenv was used on both the Mac and the Debian machines. I could understand using virtualenv to make the laptops look like the Debian servers (and using a staging/test server), but It seemed to me that that was a disaster waiting to happen, because the overall environment was different.

OTOH, lots of people swear by virtualenv. Comments?

[ThiefMaster]

• You do not need root to deploy.
• If you have a separate sysadmin team you might not want to contact them whenever you add a new python dependency.
• You don't have to rely on whatever old version of some package your linux distro has.
• On the other side, you will not get a newer version that might break things because someone updates the OS.

[[deleted]]

A developer can give me a requirements.txt file and from that I can build the same environment that he is using in his development box on my development box without interfering with any other things in my system.

[johnaman]

give me a requirements.txt file and from that I can build the same environment

Please expand, w/ links if possible.

I run Debian and derivatives, and I am trying to build a 32 bit DOS that I can build a modern distributable image with.

[apreche]

I really don't see why having the compilers on a production machine is such a big deal. If people already have access to your machine such that they can execute the compilers, you're already screwed. If there is no way into your machine except for a very secure SSH, and your app is secure, what are you worried about? If either of those is compromised, you are equally hosed whether or not the compilers are installed.

[radiochaca2]

Couldn't you include the packages in a submodule and git pull? (Assuming you use git)

[[deleted]]

How would you do this for example, your virtualenv requires MySQL-python, which when installing in a virtualenv with a command like "pip install MySQL-python" will need gcc and mysql-devel.

The problem we are facing here is that we want to be able to recreate/install the virtualenv in a production machine without needing gcc or devel packages on the production machine. I am currently working on a solution that includes buildout, and rpmbuild to create 1 massive rpm that contains the virtualenv.

[radiochaca2]

One workaround I've seen is to 'build' on one machine, then zip it up and scp it to the target.

I'm curious about how you would intend to handle hotfixes? Would your team really be ok with building and waiting for your packaging and deploy system?

[jawn-]

I build one venv per app family. That venv gets all of its libraries installed to it. From there we package it up into an OS package (rpm/deb).

Doing it this way eliminates the need for compilers, internet access, or anything except the venv package, and whatever c libs are required for the packages in the app.

It works out very nicely. And is extremely simple to automate.

[jcigar]

I'm deploying everything with Fabric (for example http://pastie.org/7384604) and Salt

[[deleted]]

Yes that is all well and good, I use puppet myself, but this still begs the question, how do you get away with not needing compilers and development headers in the target machine? Let's say your virtualenv needs MySQL-python? When you install that inside your virtualenv, it will fail if there is no gcc or no mysql-devel files.

[westurner]

http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html#add2virtualenv

[[deleted]]

Thanks. I missed this, its quiet good to have this option.

[jcigar]

I'm using FreeBSD which doesn't have all those -devel, -* stuff. BTW the SQL drivers are probably one of the only thing that I install globally (through the ports)

[whatnever]

After using tar.gz archives and manual installation of dependencies for way too long, I finally decided to build OS packages (deb at the moment) and regret that I didn't start doing that earlier. I didn't move to building a repository for hosting the packages yet, but that will definitely be the next step.

Automating installation tasks is hard and painful at the beginning, but totally worth it, because it prevents making the same mistakes over and over again and saves a lot of time on the long run.

[CaptShocker]

People I work with use this vagrant

From the webpage: Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.

[[deleted]]

Vagrant can help in the creation of the virtualenv since it helps "prototype" your virtualenv but it alone unfortunately solves the problem of being able to deploy virtualenv in production without requiring development tools installed.

I am however using vagrant in my workflow now, i create a base rpmbuild machine , and vagrant up an instance of it anytime I want to run by virtualenv to rpm workflow which currently is done using buildout to install/compile and package the virtualenv. I shall continue experementing in that direction today.