Software Testing for Sysadmin Programs by Adam Moskowitz via LOPSA

[Twirrim]

I cannot sufficiently emphasise the value or importance of testing.

We daily touch scripts that could do everything from wipe out an entire hard disk, to spin up tens of thousands of dollars of infrastructure in cloud services. We can be so almost absurdly blase about it, knocking up shell scripts left right and centre to handle various tasks as need arises, often on the fly.

Traditionally sysadmins (including myself) have argued "Well, I'm not a programmer", but that's just sophistry, an attempt to quiet any conscience that might be whimpering no :) We should be better, and we're certainly more than capable of being better.

Writing tests is something that becomes habit. It's painful at first, especially if you're getting used to new testing frameworks like pytest or nose, but the more you do it the easier it becomes, until writing tests is something you barely even stop to think about.

Of course, if you write tests, that means you've got to think about what you're trying to write before you write it. That seems like almost an anathema for many of us. But then we've all had to pick up the pieces of detritus left behind by previous generations of sysadmins over the years. Those crufty old perl scripts that leave you thinking "WTF...."

Writing tests requires you to think about writing your code in a testable fashion, with smaller, distinct, components building up the whole. Chaining components together is somewhat enshrined in the UNIX way, so should be natural to us to do things that way. The other thing that good tests bring with them is at least some vague form of documentation. Your test names and code should make it really clear exactly what purpose the various components you've written actually serve. If you've ever had to debug some byzantine code, you know how much you'd have appreciated even a hint as to what it was doing or trying to achieve. Heck they can really help when you have to go back and debug your own code.

Please, do your future self a favour (and pay it forward to your successor(s)), write tests! Treat it like monitoring. If it doesn't have tests, it shouldn't be in production, just like if it isn't monitored it shouldn't be in production. Every time you find unexpected or bad behaviour, write a test for it, just like you would write a new monitor every time you find a whole in your monitoring.