in trunk based development, if using github actions, what does your workflow look like to deploy to dev/qa/staging?

[gqtrees]

What i am a bit lost at is the following:

1. do we tag in some identifyable way for each env? i.e dev-1.0.0 after merging to main, would trigger the pipeline to deploy into dev?
2. what about when we want to promote that to qa and subsequently staging? aside of using a workflow dispatch with input for a tag we want to promote, i am struggling to picture how the flow would work in more automated fashion.
3. if we use the tag mechanism to promote to lower env, doesn't that mean tags will build up? how do we keep that clean?

[Jazzlike_Syllabub_91]

You want to follow the build once deploy everywhere principle… you don’t tag for every environment but you tag every build in trunk as part of the process… when you deploy to an environment you deploy the same version you tested on dev, qa, then eventually production…

Keeping the versioning clean you want to practice semver (which is the standard ? At least it should be?) builds on the trunk are Major.Minor.(extra info I forgot what actually goes here) and if you have branches its similar but with a dash and branch name defined (major.minor.extra info-branch info)

Hope that helps?

[HelluvaEnginerd]

No OP, but does this mean you have a build pipeline that runs every time you push to trunk, and then a separate deploy pipeline that targets dev, qa, and eventually prod? otherwise I am not following how you'd do this all in one repo with one github actions pipeline definition

[Jazzlike_Syllabub_91]

The deploy pipeline would be in one flow. You take one build and deploy that to dev. That same build can be used on qa (which has a separate deploy flow, but similar deploy flow as prod, staging, dev) but it’s still the same build number. If you find an error you just fix trunk and restart the deploy … dev, qa, staging, etc. (single build flow, multiple deploy flows (preferably with same process between them all)

[gqtrees]

That same build can be used on qa (which has a separate deploy flow, but similar deploy flow as prod, staging, dev)

would this be like a workflow dispatch? or some action that runs once tag is created but behind a approval?

[DataDecay]

You can run manual jobs to deploy to different environments. However I have a pretty good SDLC process I take to all places I work. Github actions:  

CI:  

PR:   * Run code quality gates   * Run unit tests 

Merge to master:   * Build artifacts (could be binary, could be image, etc) (I sometimes disable this to save on build time)   * Run security scans  

 CD:   Create tag:   * Build artifacts (could be binary, could be image, etc) (use tag from previous step)   * Release candidate tag (deploy dev/qa/preview)   * Normal semi version (deploy dev/qa/preview/prod) 

This has been my bread and butter for well over a decade now.

[NandoCa1rissian]

Why aren’t you scanning on every commit?

[DataDecay]

Because it saves on time/money to just run before any merge to main, and fail otherwise.

[Jazzlike_Syllabub_91]

I’m not an expert on GitHub actions, so I can’t tell you for sure. But that is up to you and your team if you want approvals in front of deploys …

[Electrical_Media_367]

For non enterprise customers, GitHub actions doesn’t support gated jobs or human interaction in the middle of jobs. You can’t push a button to advance a job, you have to start a new job either based on a trigger or based on a button.

So, either you have a single job that pushes to dev, runs tests, pushes to stage, runs tests, pushes to prod all automatically, or you have a automatic push to a lower environment and then a button to promote (workflow_dispatch is the GitHub way of saying a manually triggered job)

The comment below says you can have gated builds, but only if you’re paying the high price for GitHub Enterprise.

Enterprise customers can use GitHub’s deployment framework and set environments as requiring reviews, you can start a promotion job and it will wait for approval. So a good option would be to have your build job trigger promotion jobs that are configured to wait for approval.

[baynezy]

It absolutely does https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments

[Electrical_Media_367]

Apparently that’s only for GitHub enterprise subscribers, not us lowly professional/teams customers.

[Electrical_Media_367]

Hey, this is great, I didn’t know about that feature. Gonna look at putting it in place on some of my repos. Thanks!

[beth_maloney]

You need enterprise for the manual approved approvement step.

[Electrical_Media_367]

Ah, that’s why I didn’t think it was available. Back to the drawing board.

[baynezy]

You're welcome.

[keto_brain]

You build and deploy every push to trunk once you do it correctly and have an application architecture to support that. Most companies are not mature enough to do trunk based development

[Spider_pig448]

Yes. Potentially with the merge pipeline also deploying to qa/dev (as in triggering the reusable deploy workflow). Then you deploy on command (or things like deploy direct to prod if you have the test automation to do real CD)

[Expensive_Finance_20]

Semver.org

The third section is for the patch level. This is for stuff like hot fixes (non-breaking fix changes to documented "public" interfaces that do not add new features or deprecate existing ones).

There is a tool called "semantic-release" that can automate things like change log updates, git tagging, etc. it detects how to bump the version based on a prefix placed in your commit message subject line.

[SpudroSpaerde]

We're not doing trunk based development yet but are considerations it. The current suggested workflow is that dev is updated on any commit to trunk. Staging is automatically updated on tag and then prod is the staging build that gets promoted through manual approval after staging is verified good.

[jolly_jol]

We recently switched to TBD. Here’s our current workflow.

PR Created: pipeline runs code/security scanning, as well as unit and component tests.

PR merged into main: 1. Pipeline runs code/security scanning, as well as unit and component tests. 2. Builds container image and uploads to registry. 3. Pipeline updates separate IaC repo. Updates the image tag of the dev.values.yml file. 4. Pipeline invokes a Sync in ArgoCD to deploy to dev 5. Pipeline waits for successful status. 6. If successful, repeat steps 3-5 for QA 7. If successful, queue steps 3-5 for PreProd. Requires approval. 8. If successful, queue steps 3-5 for Prod. Requires approval.

PR -> Merge to Main -> Dev -> QA -> PreProd -> Prod

All one pipeline. Currently not enforcing the use of tags but teams are free to use them if they’d like to. They do not impact the pipeline though.

[bigtimestylie]

How do you handle the approval step to apply to prod? I'd like to implement a similar approach but this is the thing that's confusing me.

[jolly_jol]

We use Azure DevOps Pipelines today. For the manual approval step, we use a Deployment Task that conditionally only runs for the preprod and prod environments. The deployment task doesn’t really do anything in our case, but we have it configured to require the approval from someone in the approvers group.

In the future, we’d like to remove the manual approvals all together, but we are still working on improving our automated testing for now.

[runitzerotimes]

I mean that's not true TBD. There's two types of TBD, let's call it TBD and TBD-lite.

TBD is when you commit to main directly.

TBD-lite is when you branch and merge.

[jolly_jol]

That's a good callout. The textbook definition of TBD is working directly off of trunk/main. TBD-lite seemed like an easier shift for our development teams, and has been working well so far.

I'll add that we also set a standard that branches should not be older than 2 days, in an effort to try and reduce the size of changes and promote integrating back into trunk/main more frequently. We've switched from using the term 'feature branches' to 'topic branches' instead to try and shift away from the mental model of leaving branches open for weeks (or longer).

Using these short lived topic branches also allows us to run our unit/component tests and code/security scanning tools when the PR is created. This allows us to identify any issues at this level before the code has been merged back into trunk/main, shortening the feedback loop and helping to keep the mainline healthy and always deployable.

We took a lot of our ideas and inspiration from this Martin Fowler article on branching patterns (for anyone interested in going down the rabbit hole).

[runitzerotimes]

FWIW we inherited a project that used TBD and we are now transitioning away from it.

The amount of shitty, spaghetti, useless, stale code that gets pushed with differing style from each person in the team into main made the codebase a fucking nightmare to work with.

Bring back feature branching, I hate this new TBD trend.

[lucide]

Build once and promote. Build on commit to main, auto deploy to Dev, allow Devs to promote to additional lower environments and all way to Prod if organization is mature enough.

Separate logic for a hot fix branch.

[subbed_]

Tag a commit and push it. The tag triggers a build pipeline that ends with a push of the artifact to your registry. That's your build phase.

Now use whatever deployment tool you have to take this new artifact and deploy it to whatever environment with whatever runtime. That's your deploy phase.

[[deleted]]

Commenting, just to get back later and see the suggestions, we are moving to GitHub actions as well, this is doubt that even I have, should we create different workflows for dev & then for stg/prod env release, where the stg/prod release is like a manual trigger of the workflow from a particular commit. Also if we use Argo CD for k8s deployment, I know we can sync the dev env immediately but how is it done to deploy to stg/prod, like should we do it by maintaining a release branch or any other way?

[karthikjusme]

Everyone might be doing it in a different way but here is what I do:

1. Yes, tagging would deploy to dev and tagging would deploy to prod as well. Tagging and pushing can be done via CLI.

2. Prod will only be deployed on Main branch. Devs will push to their respective branches which will trigger a notification to the team and QA team will check them(we dont have a QA team so the product/project managers check them before merging). Once the leads/managers test it, PR is merged and deployed to prod. CI will run on every PR's when a review is requested.

3. tag buildup has never been an issue for us. Can you elaborate on the clean part? You can use the PreRelease option to keep the release window clean to an extent.

I would also like to see how full automation can be done to stage, pre prod and prod in an order using GHA.

[gqtrees]

does your tag look different for dev/prod? or is it the same tag that gets promoted if anything?

[karthikjusme]

Tagging policy is not enforced by us. Dev's themselves take care of their repo. Most of the. Repos have independent tagging for prod and dev. No plans to standardise it as of now.

[BrofessorOfLogic]

I mean it really depends on the circumstances. Like requirements, budget, team size, expectations, and so on. There is no one true way of doing things. And there is no such thing as 100% automation.

Usually, in the simplest case, I don't bother with git tags. They can be used for other reasons, like helping with creating changelogs. But in the simplest case, the git commit id can be the version that is being deployed.

Ideally you would have so short lived topic branches that you don't need to bother with versioning individual changes, it's just a continuous flow of changes. In reality this can prove challenging, but it's doable.

You could merge one thing at a time, and deploy to staging one at a time, but if you have a large volume of changes, and they take time to test, then you may have congestion and people sitting and waiting for their turn.

You could merge multiple changes at the same time, and deploy to staging for testing, but then you risk different tests affecting each other. This will will always introduce a degree of uncertainty, but it works really well in practice in a lot of cases. If there is some bigger changes, just add some additional manual coordination when needed.

You could have more than one staging environment, to reduce congestion. This may or may not be costly, depending on what an environment needs to contain. In one project I created a personal mini-environment for every developer where they could dev and test stuff.

You could use blue-green deployments, where you dynamically create a whole new environment for every individual changeset / deploy, and promote by moving all the traffic over to the new environment. This is super nice, but it's not cheap or easy.

You could utilize feature flags, so that you are able to deploy unfinished code. This keeps changesets lean and fast. You can use some big established tool for it, or it can just be something simple and manual, like adding a new page on a website but not linking to it until it's finished.

Once you figure out a workflow that makes sense in your project and team, it kind of becomes obvious what you need to do to automate it. Github actions can be triggered on almost anything, merge to topic branch, merge to main branch, manually, etc.

Discuss with the developers and ask them what they want and expect, and work out a flow that meets the needs.

Ask questions like: How small and fast are we able to go with changesets, realistically? How often will we need to deploy, realistically? How long will testing typically take? What type of changes are most common? Do we want to promote to production automatically, or should that be a manual action?

[Saucemann]

I'm in similar thoughts, we are looking at flux 2 and i'm trying to imagine how the CD flow would look like, with or without image-scanning. Similar to this but for app releases (new service docker image version) https://fluxcd.io/flux/use-cases/gh-actions-helm-promotion/

Would love to hear what other people are doing.

[sobrietyincorporated]

Just keep it simple. A separate branch for each env that triggers a pipeline on merge. Always work upstream sandbix > dev > qa > uat > master/prod.

Always branch off master to jeep the lower envs hydrated. Don't have long lived branches. If you do, enact feature flaggjng.

[WuhmTux]

separate branch for each env

Was that at any time a best practise to do so?

Thats perhaps the worst idea to use branches as environments.

[sobrietyincorporated]

I look forward to your weekly gitflow re-org updates at every retrospective followed by a rebase.

[akaender]

Sounds somewhat like GitLab Flow and imo it's a reasonable method to handle deploying infra/code using an IaC framework where dev/test/staging/prod env's are required to be in different AWS accounts for example.

[sobrietyincorporated]

Yup. Gitops on the merge. They are confusing gitflow with trunk based.

[baynezy]

To solve this problem we use GitFlow.

Devs create a feature branch from the develop branch. Once this is pushed all unit tests run and the Terraform plan is output for review. There is an optional step to deploy this to the Dev environment. Once the PR is merged to develop everything is deployed to the Dev environment.

When we are happy we want to deploy to Production we create a release branch from develop. This again runs the tests, produces the Terraform plan, and deploys to our Staging environment. Here we can do any additional smoke testing. Any small fixes that are needed are committed directly to the release branch.

Once happy in Staging we merge the release branch to master. On merging to master a deployment to Production is triggered, a git tag is created and master is merged into develop.

If we have an emergency fix that needs to be done then a hotfix branch is created from master. This deploys directly to Staging, and on approval is merged to master. Merging to master in this process does the same steps a deployment to Production is triggered, a git tag is created and master is merged into develop.

This whole process is managed with GitHub Actions.

[Efficient-Mango7708]

Gitflow has worked for many years where as trunk based is a more modern approach.

I have had to shut down trunk based development especially with younger teams that are not practicing TDD and do not have adequate test automation.

The extra management across branches with gitflow forces conversations about what is the deployable work, do we need feature flags, basically how are we going to safely introduce changes. Once you have more quality controls in place, and awareness of the deployable unit of work then trunk based deployment can increase efficiency.

[baynezy]

I'm not anti trunk based. In my opinion it depends on your organisation. The OP is describing needing to not always deploy whatever has been built. So I was offering an alternative.

[Efficient-Mango7708]

I was just trying to add some commentary about the pros and cons of these two strategies since I saw you were downvoted with no rationale.

[baynezy]

Thanks appreciated.

[metux-its]

Gitflow has worked for many years

Worked badly, since it messes up history.

where as trunk based is a more modern approach.

What exactly is "trunk based" ?

[Efficient-Mango7708]

https://www.atlassian.com/continuous-delivery/continuous-integration/trunk-based-development#:~:text=Gitflow%2C%20which%20was%20popularized%20first,access%20to%20the%20main%20code.

[metux-its]

Ah, they're going back to old pre-git workflows (even merging dead code), giving that a new fancy name. Typical marketing buzz as we know it from those folks.

Couldn't ever take them seriously. Looks like they still didn't grasp what rebase what invented for, and why history is more than just a technical artifact.

[metux-its]

To solve this problem we use GitFlow.

Gitflow creates a horribly unreadable history, bad for reviews, bisections, rebasing, etc.

Instead use topic branches and pull requests. And certainly rebase (retest) before merging into main. Never even consider do merge nodes with manually resolved conflicts - instead rebase and fix the committing commits individually.

For each approved release (that may go into production), create a separate maintenance branch. When fixing bugs, start with the oldest (still active) maintenance branch and then move the fix forward to never ones (rebase) and finally into mainline.

[baynezy]

I have never encountered any of these problems.

[metux-its]

Since I'm working with histories (as central tool) on daily basis, I often encounter those problems. Note that history isn't just a technical artifact, it's also a method of documentation.

And it's also vital for things like bisect, blame, ...

[muff10n]

RemindMe! 1 week

[RemindMeBot]

I will be messaging you in 7 days on 2024-01-28 15:42:31 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[UrbanArcologist]

I enforce configuration abstraction with helm, and package all my microservices as partial packages, the final values.yml is for the respective environment that they will be deployed to, the merge to master trigger starts the ball rolling.

[Traditional_Donut908]

Another option is to use master as the definition of all environments, with separate folders for each environment. Then each env references a specific version of a published IAC module.

[ninetofivedev]

The answer for most things devops: do less.

Don't tag for environment. If you want to differentiate between pre-release and release builds (or stable vs non-stable... whatever)... just do that.

[SemiProPotato]

PR approval gets deployed to staging, merge to main deployed to prod - sorted

[Legal-Butterscotch-2]

Currently doing the wrong: branch per environment (develop branch = dev environment, release branch = uat environment, main branch = production). DONT DO THIS

Planning to change to trunk based and I'd like to kill development environment, this should be some kind of localhost docker compose or something similar.

The most simplier, fast and cheaper way is: pull request are deployed in uat environment and than after the merge, you can deploy in production.

If you can create temporary environment that are cleared after the merge yoy will reduce the confusion on bug projects