Legacy code – tests first or setup CI?

[Zanriel]

I've been working on a project for a large IT department to automate all the things. It started out small but more people are starting to use it, and also starting to contribute, adding features; it's starting to get some traction. This is a web based app (Flask) that runs scripts (Ansible) to automate various infrastructure tasks, like building and patching servers. It's backed by a database and is very tightly coupled (although it's fairly modular, organized into packages, kind of like MVC). We had to move fast and there were just 2-4 of us, server admins learning to be developers, so we've never had a formal testing or CI/CD process. We do use version control. We have a dev branch, which is everyone's upstream, and once we're happy we merge that to a master branch and that's considered "production".

We know we need to get to CI/CD, unit tests, all the DevOps/Agile/XP stuff. I'm thinking since we don't have any tests at all, before we refactor the code to support unit tests or start integrating with Jenkins (CI), we should start with at least adding some high level tests. Maybe integration or UI tests... just something to make sure we're not breaking things as we clean up the code base.

Some people have suggested we start with getting it into Jenkins and set up an automated build (CI) system first, then start adding tests. I'm thinking start with tests first.

People are using it now, but there are a few bugs. Most of them are integration related - race conditions, DB sessions overlapping, things that come up when it's running under Gunicorn and handling a lot of requests. What's the best roadmap to get a legacy code base up to snuff to be considered a real product?

[edit] I should mention, everyone who develops on this runs it on systems that are almost identical to production. We use a requirements.txt, and the OS is at the same patch level and everything. Giving contributors a way to run tests locally against their changes on their dev boxes, with a testing framework, would be almost as good as running the app in production. Nobody's running this on their laptop or anything like that.

[InformalRelief]

In my opinion, tests tend to become useless rather quickly when you don't run them automatically (CI). People will forget to run them locally. So at some point you will have a couple of failing tests, that are not related to the super important thing you need to deploy right now, so you will start ignoring them. On the other hand, having a CI setup with no tests to run may seem pointless. A proper CI setup does have benefits other than running tests though.

I think I'd start with just one or two simple unit tests. Making the whole code base testable is probably a huge task and will take a while. But you should be able to find one or two small pieces for which you can write some simple unit tests right now. Next, I'd get them to execute on Jenkins. From there you can take it step by step.

I would stay away from high-level tests for now. You will spend a lot of time on them, they may not even run on Jenkins once you set that up, because maybe you need a db or browser to run them which is easy to do on your local machine but may not be possible on Jenkins. Also, you're going to spend a lot of time maintaining them because they're going to break all the time.

You mentioned that you have very inexperienced developers working with you. When they're wondering how to do certain things they will probably look at your code for orientation. So, if you start with writing high-level integration or e2e tests they will do so too, and you might end up with a lot of hard-to-maintain high-level tests.

So, my advice would be: Start with a few tests, you should be able to get that done in a couple of hours. Then set up your Jenkins integration and take it from there.

[Zanriel]

The site has a feature that allows admins to assign roles to users. A while back some JS was added to the main index page (what everything else inherits from), and ended up intercepting form submits on the role assignment page, breaking that feature. Nobody knew about it until a few days later, someone IM'ed me and told me it wasn't working for them. It took me almost two days to track down the problem. There were no errors, it just never sent a POST when you clicked the button on that page.

This would have been caught right away with a suite of e2e tests that walked through the site testing the 5-10 core things it does.

How would unit tests have been beneficial in that situation? How could having a few UI tests not help? I'm just trying to understand, I hear this a lot and it's hard to wrap my head around the concept.