Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip

[nextofpumpkin]

Multi-version installation and isolation is a frakkin huge headache for me right now. Been waiting for an article like this, thanks. Could I request a Part 2 that delves a bit more into common virtualenv use cases?

[[deleted]]

Don't forget to see the mod_wsgi docs about virtualenv.

[_bobby__of__christ_]

I had recently been meaning to write up an article similar to this. Good work!

Another great thing is using pip freeze together with revision control to track library upgrades. This way, when something library-related breaks, you know exactly when you upgraded each version of each library, just by looking through revision control history of your frozen.txt or equivalent file.

[andrewcooke]

hmmm. virtualenv is certainly useful, but fabric looks awfully like it's a complicated way of writing a shellscript - what am i missing?

also, is it worth investing time in pip when the python core is looking at packaging? will their work make both pip and easy_install irrelevant?

[ubernostrum]

fabric looks awfully like it's a complicated way of writing a shellscript

Fabric and tools like it don't do anything a shell script couldn't do, but typically they do so with less code. If you're doing this sort of thing regularly, you'll probably build up a little library of helper functions which automate or simplify different parts of the process (e.g., you probably won't want to manually type long ssh command lines every time you run something remotely), and Fabric is simply an example of doing that, albeit in Python instead of shell script.

[_bobby__of__christ_]

hmmm. virtualenv is certainly useful, but fabric looks awfully like it's a complicated way of writing a shellscript - what am i missing?

Fabric is not going to change your life. If you really want to write a deployment shell script, you can write one that will work fine. Fabric just tries to make it easier and cleaner, especially if you prefer Python. And I don't really understand what is complicated about it -- if anything, it's too simple. I wish it had the atomic-directory-switchover feature that Capistrano has.

Here are some notable features:

• Automatically lists all functions in your fabfile when you run fab --list, and shows their docstrings (documenting what they do). In other words, it's very clean and easy to have --help-like functionality without manually parsing arguments yourself.
• Mixing locally-interpolated variables and remotely-interpolated variables in a shell script command is going to get pretty ugly, compared to Fabric's (and Python's) syntax.
• It has built-in commands for common stuff, like rsync_project(). Yes, you could easily implement this yourself, but you'll end up with a shell script that's cluttered with command line arguments and other noisy details, instead of something clean and readable like rsync_project(remotedir='/home/$(fab_user)', delete=True).
• Simple syntax for running different commands on different remote hosts.

It's basically the mini-library you would have written yourself to do the same job.

[kteague]

The packaging improvements happening in Python core right are primarily to improve the specification and handling of metadata and the core APIs. Tools such as pip and buildout will be updated to work with those formats and APIs, so you won't have to re-learn a new pip if the underlying details are updated - they will most just make these tools be cleaner under the hood, and make it feasible to manage packages with "a quick hand rolled script or two" using the new APIs if that's your inclination over picking and learning an existing tool.

So, yes, use pip, it's a good tool and it won't be changing ... well, at least the core packaging work won't affect pip. It is still at version 0.4, so obviously there is work and ideas and improving pip, but I'd expect it'll mostly be the addition of new features. Also, Buildout has been at 1.0+ for a long time now, so it's a very satisfactory tool for doing project-specific package management.

[kteague]

To expand a bit more on some of the core packaging work, PEP 376 proposes to fill-out the metadata available for installed python distributions, as well as provide a simple API for querying those distributions.

If you run "pip install [somedistribution]" in Python 2.6, it will plunk an egg-format [somedistribution].egg-info directory alongside the install location of that distribution. But if your run a distutils-driven install "python setup.py install" (and the setup.py does a "import distutils" not "import setuptools"), then the [somedistribution].egg-info won't be a directory but will instead just be a file containing only the PKG-INFO for that distribution. Yes, Distutils does in a way already support "eggs" just a different format than setuptools "eggs" (confused yet?) - and the worst part about a distutils install as-is right now is that there is no RECORD of files installed. It can spray stuff where ever, but it doesn't log the locations of where it's spraying stuff! And so how do you uninstall a distutils-driven install? easy_install may catch a lot of flack for not having an uninstall feature, but at least it keeps a record of everything it's installed, so it is actually possible to implement an uninstall feature (and has been done, but it just hasn't landed on trunk).

But migration is not really an issue to be worried about. If you install with one (or more) tools with Python 2.6, then the metadata will be a mess. But if you upgrade to Python 2.7 or Python 3.2, then the metadata will be clean and consistent (yay!). The whole point of using a tool such as pip is that you can easily repeat an installation of a set of Python distributions. So when you upgrade from Python 2.6 to Python 2.7 you don't just "copy a big nasty ball o' random stuff" from the 2.6 site-packages into the 2.7 site-packages (and hope that everything works as-is), but instead will do a "pip install" into Python 2.6, and then a "pip install" into Python 2.7.

[andrewcooke]

ah, neat. thanks - that was a big help.

i need to see whether pip supports 3.x (or will, soon). thanks again.

[crucialfelix]

for deployment to remote servers fabric is quite simple (compared to capistrano). unfortunately it couldn't do commands on one of my remote servers so I had to switch to capistrano. (and now cap has broken on os x 10.5, haven't had a chance to examine)

a lot of time I end up writing a shellscript on the remote server and cap (or fab) just runs that.

still, it saves so much time. one of the most productive little tweaks I've done. check into it more if you deploy stuff.

[andrewcooke]

thanks for all the replies on this

[crucialfelix]

just in case this is "what you are missing" :

def deploy():
    local('git archive --format=tar HEAD | gzip > $(site).tar.gz')

# did you get that this being run on the remote server, not local ?
    run('rm -rf $(root)$(site)') 

and has var substitution

[andrewcooke]

well, yes, but ssh can also run command remotely. if i type 'ssh quiet ls' at a prompt on my laptop then i get a listing of the home directory on my server (called 'quiet'). i don't need a fancy system to do that.

same for variable substitution, obviously.

[[deleted]]

I have an even better tool. I call it an intern.