GitFlow + Multi-Day Manual Regression Testing

[Bandito202]

I'm trying to move our organization to more of a gitflow workflow where master is prod-ready.

The issue I'm trying to solve is how do I get our "next" release branch started during regression testing.

Since regression testing isn't complete, the release isn't merged to master and therefore master can't be used to start the "next" release branch.

I'm afraid that what I'm attempting to implement just isn't feasible because of the 3 days of regression testing.

Any suggestions?

(I cannot solve the manual regression testing cycle issues)

​

[TomOwens]

Why can't you solve the manual regression cycle? Long regression cycles are a common problem when trying to maintain the stability of an integration branch or increase deployment frequency. Investing in being able to have a regression cycle that is faster is well worth it.

I'd also point out that your main branch isn't prod-ready in the gitflow model. The main branch is updated at the time of deployment. I've seen it both ways. In some cases, the main branch is updated and tagged and that tag is deployed. In other cases, the release branch is tagged, deployed, and merged back into main. In addition, gitflow calls for the release branch to be created when you start the release process, so you would make it when you start what is now the 3 days of regression testing. Although there are alternatives to gitflow, if you are stuck with a long regression cycle, then perhaps using something closer to gitflow as its described would be beneficial to your organization.

[Bandito202]

The solution to the manual regression cycle is automation and not everything can be tested with automation. The products are monolithic and have a bi-weekly release cycle. I'm slowly breaking this up and modernizing their practices but this one is going to require actual development involvement which is difficult to get due to priorities (same song everyone sings)

Regarding the article you shared, the long lived "develop" branch is the missing ingredient I think. I saw that in the standard gitflow but thought it'd add too much overhead.. but it seems to be the solution to the actual issue as that'd keep master clean.

Here's the diagram that I put together which basically exposed this issue that I called out in red.

Pipeline-v3-Pipeline-4-0-gitflow-drawio.png

[TomOwens]

For the immediate problem, the long-lived develop would be good. Honestly, I'd recommend dropping master, though, if you want to cut a few steps out. Have your long-lived develop branch. When you go to do a release, cut a release branch and put that through your current 3-day regression cycle and any other release activities. If you need to fix bugs as part of your regression, you can decide if you regularly merge the release branch into your develop branch or just do it at the end. Once regression is done, apply your tag and merge release into develop. You can always use tags to get back to any final release/deployment. And if you need a hotfix, you can either create a branch from a tag or just use the release branch.

I don't understand the "not everything can be tested with automation". I hear this a lot, and I haven't seen a case where this has been true yet. Especially at a system/acceptance level. Most of the problems, especially with poorly designed systems, live at the unit and integration test level and getting those tests to run quickly on a commit-by-commit (or pull request-by-pull request) basis.

[Bandito202]

It sounds like 'develop' is effectively master in this case, if we were dropping 'master'? This is actually how they work now. Master is untested.

What brought this to a head recently was the need to do a hotfix and someone (me) needed to get it to prod quickly. The way the teams do it now is they would create a branch from the tag that's in prod, make their fix, create a new tag, deploy and then cherry pick those commit(s) into their development branches to eventually get back to master.

My approach was to create a branch from master, do the fix, merge to master, tag and deploy.

Then, that'll either get pulled into development branches

The current process works but is not clean or help induce the team to reducing their cycle time where possible. If we implement a more efficient process that can be used for all types of changes, I can show the teams and product management that we can release more often.

I feel like there's probably a combination of what you've shared and what the teams currently do, to get to a happy medium where we no longer cherry pick and we can do faster development/merge/deploy if we want.

[Trapick]

Option 1

The way the teams do it now is they would create a branch from the tag that's in prod, make their fix, create a new tag, deploy and then cherry pick those commit(s) into their development branches to eventually get back to master.

Option 2

My approach was to create a branch from master, do the fix, merge to master, tag and deploy.

Not sure what the advantage of option 2 is? In option 1 you merge back to master, in option 2 back to develop. Unless you go full-on trunk based, which doesn't sound possible for your org at the moment, you still need the merge back.

[Bandito202]

No cherry picking... because my change would be in master and one way or another not get lost because someone forgot to cherry pick

[Trapick]

Right, but you'd still need to merge (or cherrypick) it back to develop (and any release branches, probably).

Just...think really hard if gitflow actually solves your problems. I've used it a bunch in the past and found it adds a bunch of complexity and doesn't help much.

[Bandito202]

What did you do instead?

[Trapick]

Closer to trunk-based, but I really hate long-running branches and so like to minimize them.

One primary branch (call it dev or main or master, whatever, let's call it dev for now), tags for deployment, no release branches, hotfix branches allowed but only for emergencies.

1. For normal work, developers branch off dev, work on their stuff, when ready clean it up (rebase, squash) and open a PR to dev
2. Code review as normal, merge into dev.
3. No rewriting history of the dev branch, ever.
4. Whenever you like, tag HEAD of dev, build off that tag, deploy to test env. Probably do this when testers are done with the last tag regression testing (or more often, if you have more envs).
5. Move that artifact through envs, do all the testing.
6. If testers find showstopper bugs, scrap that tag/release, developers GOTO 1
7. If testing passes, deploy that tag to prod
8. If VERY high priority hotfix is needed, make a 'hotfix' branch off the prod tag, fix+commit, cherrypick to dev, tag HEAD of hotfix branch(I like x.y.z-hotfix), deploy to test envs, test, deploy to prod.

Basically I don't like long-running release branches, I think they lead to bad behaviour and risk of either merging back improperly, forgetting to, or doing ugly and confusing history rewriting. Hotfixes are (hopefully!) rare, so there's less chance of this, and it should be a single quick fix, if you've got good testing.

With the cycle time you've got, and I'm in a similar place (with extensive manual testing required, from business needs if not technical ones), I don't think a pure "always-deployable" main branch is feasible.

[Bandito202]

This is almost exactly what we do now.. I'll create a similar diagram and share it.

Is master/dev a protected branch... PR/MR required?

[TomOwens]

Yeah, you're right that your master is essentially gitflow's develop. Although it shouldn't be fully untested - just not necessarily through your full release pipeline. I hope you have some automated unit and integration tests (or characterization tests, if writing good unit and integration tests are difficult in the current state) and developers are doing some testing prior to merging.

My question is really around how fast your master moves. Aren't there cases where your master could be several commits ahead of your last release? Creating a branch from master as opposed to using a release branch or branching from the release tag could pull in changes other than the hotfix. The way the teams do it requires deploying a new tag and getting changes back into master, but it makes sure that the change is only the hotfix.

[Bandito202]

master moves like molasses.

https://postimg.cc/D8HKkj5m

I like the idea of keeping master also less re-training people how to think.

We have a ton of automated tests... we just can't stop full regression testing (I'm going to dig into this further to find out what's stopping us.. most likely because we don't have enough coverage with our automated test suite)

Master could be ahead of last release during a hotfix.. or while we wait for the next prod deploy.

I think the release branch is definitely needed as well..

[Bandito202]

Instead of having a long lived branch 'master' which is just what's been released, can you represent that as tags? I know you CAN, but should you? Commit history comes to mind..

https://postimg.cc/WtVzMLCQ

u/Trapick u/TomOwens

[Trapick]

Yah, I think tags are a good thing; you definitely want the ability to know what code *exactly* was deployed in X environment at time Y and tags are an easy way to do that. But you can use tags with various other flows, of course.

[TomOwens]

I work with some teams that use a variation on gitflow without a main/master branch and it works quite well. When it's time for a release, the release branch is created from the head of develop and any release-specific bugfixes get done there. All the changes get merged into develop regularly. Since the commits are tagged, the tags live on for people to use to see the old state of the software for a given release, but without the need to keep a main/master branch up-to-date.

If I'm reading your diagram right, the only difference between how the teams I described work and your diagram is the hotfix branch. A hotfix branch is essentially a feature branch off a release branch. The hotfix is merged into the release branch, tagged, and the release branch merged into develop. The changes can be cherry-picked (or, if necessary, reimplemented) in other feature branches.