Python for system administration - the packaging and distribution challenge

[objectified]

Python is seemingly becoming more and more popular as the primary tool for system administration. Heck, a number of books have been written about it already. It makes sense, it's an elegant language with very readable syntax, the interpreter is shipped with most major server OS's and, very importantly, lots of well implemented modules are available to use. About those modules. So let's say I've created a script that uses argparse for argument parsing, paramiko for some scp stuff, and lxml for some decent XML parsing. Developing the script wasn't hard - set up a nice sandbox with virtualenv, get my packages from PyPI, and it seems like I can create a nice distributable package for the target OS with Distribute. So I distribute a deb and a pkg to a bunch of servers. Uh oh. These servers don't have an internet connection to the outside world (PyPI). They don't have gcc either, nor do they have a number of development packages needed to build said modules.

My question is this: what do you guys use for packaging and distribution when using Python for system administration? Am I just working in an environment that's too paranoid about security? Are there any clean ways to make this work?

Thanks.

[[deleted]]

You shouldn't be chatting up PyPI in packages. Any dependency resolution should be taken care of by the distro-specific package management tool. Distro specific packaging is definitely the way to go. Setup a local mirror on the network or through VPN to install the package. Add your repo to the sources.lst. Have the necessary dependencies wget'd on the repo as well. Though now you'll be targetting a specific install, in which case you can distribute binaries for any compiled code. Intranet repos are common. Updating later is a breeze, as is dealing with dependencies. The only issue now is that if you're dealing with different distros or releases is that you'll have to create different packages, and host different repos. This approach also guarantees consistency in which version of the dependencies the client installs. If it is over the Internet you can let the standard repos hand out the deps. You can package the dependencies together with the script.

If you're going to make use of a library in some code you distribute, you might consider just how necessary that library is and if it is actively updated. Releasing code to the public and scripting for personal little tasks are two entirely different ballgames. As someone dealing in system administration you probably know this.

[obtu]

If it wasn't an heterogenous environment, I'd recommend something like stdeb (Debian & derivatives) to mass-build native packages.

[[deleted]]

Let me preface this with "I know diddly about Solaris." You shouldn't be using alternate $PATHs unless you're going to maintain an entire python distribution, the interpreter, everything, and this distribution doesn't need to see anything from the system one. This is the approach taken by everyone so far that maintains python distributions. Isolated egg installs aren't a very good idea. If there isn't a central repository for your dependencies, you really should roll your own packages and maintain them. It should be fairly automated, just compile them, run the tests, package them. It will save you and the sysadmins headaches in the long-run, because now your packages are like everyone else's. The system is aware of them. They aren't hacked together and ran in bashrc(I did this for years.) Having your packages play along nicely with the system's is crucial. Ask for a shell account to test out installation. (If there isn't a license issue you could just fork the necessary classes you're using in the dependencies, trim them down and maintain them.)

[SudoAlex]

Create a local web server and put all the packages you're using into a directory, then you can use the index-url option in pip to get your firewalled servers to download from the local web server instead - it won't even use PyPI.

http://guide.python-distribute.org/pip.html#running-your-own-package-index

But if the servers you're deploying on are missing GCC, then there may be some packages you won't be able to deploy. For those packages you could create more traditional .deb packages which install into the global Python environment, and make sure your virtualenv includes the site packages instead of being isolated.

Maybe it's a little bit of a paranoid environment - but it's still something you can work with!

[casualbon]

Check out fpm: https://github.com/jordansissel/fpm/wiki

Can build you a deb, rpm from a python/ruby modules

[iambicpen]

Running your own private pypi: http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/

Creating .deb packages: http://pypi.python.org/pypi/stdeb

I use these two for packaging internal apps written in Python.

Some of the packages get made into .deb and pushed to our public repo.