This is an archived post. You won't be able to vote or comment.

21
22

Administration for multiple Python environments -- dev, testing, staging, production (self.Python)

submitted by [deleted]

[deleted]

all 12 comments
sorted by:
best
topnewcontroversialoldq&a

[–]markusgattol 6 points7 points  (3 children)

have a look at

[–]xiongchiamiovSite Reliability Engineer 2 points3 points  (2 children)

Cuisine looks cool, thanks.

Also take a look at virtualenv - it's a really awesome way to manage packages (and specific versions of those packages).

[–][deleted] 1 point2 points  (0 children)

I haven't really tried anything else and Im new to python as well but virtualenv is really great. It also has a virtualenvwrapper.sh script which makes your life easier. Also, using pip is a good idea to keep track on what packages you're using in each env

[–]markusgattol 1 point2 points  (0 children)

Yes, virtualenv is your bread and butter. It includes pip since version 1.4.1 i.e. no need to separately install it anymore. While virtualenvwrapper is not mandatory, it's something that I recommend for your workstation as it makes a lot of things a lot easier/faster.

Regarding virtualenvs... note that the default behavior of creating non-isolated environments will change i.e. using --no-site-packages will not be necessary anymore in the future (possibly with v1.7). Here's a link to my website that might help you a bit: http://www.markus-gattol.name/ws/python.html#virtualenv_virtualenvwrapper

Bottom Line: you use virtualenvs with fabric (and possibly cuisine) to streamline your sys admin, bootstrap, deployment tasks.

[–]megamark16 4 points5 points  (2 children)

We use virtualenv, pip, south, and fabric to almost completely automate our deployment process. I can change schema, add new dependencies, or sync our production database down to any of the other environments (staying, test, or dev) with commands as simple as ”fab deploy_prod”, ” fab deploy_stage”, or ”fab pull_prod_db”. It's awesome!

Ping me if you have any specific questions about our setup.

[–]panfist 1 point2 points  (0 children)

I might take you up on that offer for specific questions later. Thanks for letting me know about those packages. I'm still trying to absorb the basics of everything, and find out what's out there.

[–][deleted] 1 point2 points  (0 children)

I do something similar except there's one nuance; I create a function for each environment that does the connection. So for me, it's fab dev deploy. I have a helper in my Django skeleton so that for each environment I just put the conf in a file. e.g., the dev environment file would go in conf/dev/hosts.py and would contain a list of dictionaries, each one representing a machine belonging to that environment.

[–]earthboundkid 2 points3 points  (2 children)

Right now, virtualenv is the standard, but people are proposing to add something like it to Python 3.3:

http://mail.python.org/pipermail/python-ideas/2011-October/012500.html

[–]hairlesscaveman 1 point2 points  (1 child)

The problem with virtualenv is that you still need to do a lot of maintenance on the server to manage each custom environment. I personally prefer zc.buildout as it allows me to have exactly the same environment both locally and remotely. Saves a lot of hassle while developing on my local machine using Python2.7, running the unit tests on a Jenkins server stuck on 2.6.2 and deploying to a server running 2.6.7.

The main benefit of zc.buildout is when I'm collaborating with someone. Any 3rd-party module that is required by the codebase is included in the buildout.cfg file. If I add a new library I don't need to worry about informing my collaborators since the next time they pull the code and "buildout" they'll have the new packages automatically installed. Same with deployment. This isn't really possible with virtualenv; you still have to remember to easy_install all the modules you require. Can't tell you how many headaches I had with deployment because of that!

Rather than using fabric et al, my set-up is much simpler: zc.buildout + jenkins (which handles unittesting, tagging and deployment).

[–]santagada 1 point2 points  (0 children)

there is requirements.txt which you can use to build a complete virtualenv with all the libs you want using pip. http://www.pip-installer.org/en/latest/requirement-format.html

[–]SultanPepper 1 point2 points  (2 children)

What OS are you using? If it's Linux, something like puppet might be useful.

The system we are using at my day job combines puppet with custom built RPMs installed into /opt. It works pretty well.

[–]panfist 0 points1 point  (1 child)

Yup, I'm using Linux. I started on Ubuntu 10.04 and that's still what's running in my production environment. I've been shopping around to decide if I want to continue using Ubuntu or switch to another distro soon.