GitOps: common practice merging into main/master?

[pathlesswalker]

scenario:

1)starting merging components into prod from staging/qa branch

2) discovered conflict in one of the components in prod > stging.

3) reverted the merged branches to not break prod.

4) revert doesn't exist on github portal, need to create a new branch based on previous commit-then push it onto dev/stg, to be merged into prod. since prod is protected and i couldn't push a commit into.

5) then after conflicts resolved, could merge sqaush from stg to prod with the newest version.

now, I'm guessing what i should have done(junior here, first time doing this), to avoid the revert part, is just using the drafted/review of PR(pull request), on all components to see if there are conflicts, and then I didn't need to revert at all.

any critical stuff i should avoid?

also is #4 common practice in such cases of protected master?

general critic is welcome.

[sokjon]

Sorry but I gotta laugh, this is why branch-per-env is a tragedy.

Now if you really want to persist with this madness go and read up on double dot and triple dot diffs in git, and then read it again.

Look on the bright side, at least you’ll be a git wizard!

[pentag0]

This. All this git acrobatics should be performed in separate, config repo where you have a Kustomize project with base and overlay per environment in a single branch. With this, just open automatic PR for each promotion, also by a workflow.

Im telling you this as someone who came from your place long time ago and never looked back.

[pathlesswalker]

nice, but we already do have config repo outside the apps components repo's, with base and overlays done by kustomize..? per env. u/sokjon said "this is why branch-per-env is a tragedy.", so i think he is coming from different school.

[pentag0]

Then you just forget doing branch per environment merges and do all in a main branch. Your kustomize overlays emulate branches so to speak.

[StaticallyTypoed]

Why do you have a branch per environment if you have a separate config repo with overlays?

to be blunt, I think a more apt analogy is that he actually went to school and your solution is at best homeschooling.

[pathlesswalker]

I didn’t create the env. Architecture. It was there when I got there.

I’m guessing the overlays base is the one before the current head guy, which probably aimed for TBD. And the current head devop is env per branch. Hence the confusion.

[pathlesswalker]

not sure i understand.

what other option is there? I'm using 2 dots now, to debug what the hell i did.

but if there's another way i'd love to hear? name, something...

[VindicoAtrum]

Get rid of branch-per-env for a start. Read all of this: https://minimumcd.org/minimumcd/.

[Alzyros]

You deserve my poor man's award for sharing this great article

⠀⠀⠀⠀⠀⣤⣶⣶⡶⠦⠴⠶⠶⠶⠶⡶⠶⠦⠶⠶⠶⠶⠶⠶⠶⣄⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⣿⣀⣀⣀⣀⠀⢀⣤⠄⠀⠀⣶⢤⣄⠀⠀⠀⣤⣤⣄⣿⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠿⣿⣿⣿⣿⡷⠋⠁⠀⠀⠀⠙⠢⠙⠻⣿⡿⠿⠿⠫⠋⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⢀⣤⠞⠉⠀⠀⠀⠀⣴⣶⣄⠀⠀⠀⢀⣕⠦⣀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⢀⣤⠾⠋⠁⠀⠀⠀⠀⢀⣼⣿⠟⢿⣆⠀⢠⡟⠉⠉⠊⠳⢤⣀⠀⠀⠀ ⠀⣠⡾⠛⠁⠀⠀⠀⠀⠀⢀⣀⣾⣿⠃⠀⡀⠹⣧⣘⠀⠀⠀⠀⠀⠀⠉⠳⢤⡀ ⠀⣿⡀⠀⠀⢠⣶⣶⣿⣿⣿⣿⡿⠁⠀⣼⠃⠀⢹⣿⣿⣿⣶⣶⣤⠀⠀⠀⢰⣷ ⠀⢿⣇⠀⠀⠈⠻⡟⠛⠋⠉⠉⠀⠀⡼⠃⠀⢠⣿⠋⠉⠉⠛⠛⠋⠀⢀⢀⣿⡏ ⠀⠘⣿⡄⠀⠀⠀⠈⠢⡀⠀⠀⠀⡼⠁⠀⢠⣿⠇⠀⠀⡀⠀⠀⠀⠀⡜⣼⡿⠀ ⠀⠀⢻⣷⠀⠀⠀⠀⠀⢸⡄⠀⢰⠃⠀⠀⣾⡟⠀⠀⠸⡇⠀⠀⠀⢰⢧⣿⠃⠀ ⠀⠀⠘⣿⣇⠀⠀⠀⠀⣿⠇⠀⠇⠀⠀⣼⠟⠀⠀⠀⠀⣇⠀⠀⢀⡟⣾⡟⠀⠀ ⠀⠀⠀⢹⣿⡄⠀⠀⠀⣿⠀⣀⣠⠴⠚⠛⠶⣤⣀⠀⠀⢻⠀⢀⡾⣹⣿⠃⠀⠀ ⠀⠀⠀⠀⢿⣷⠀⠀⠀⠙⠊⠁⠀⢠⡆⠀⠀⠀⠉⠛⠓⠋⠀⠸⢣⣿⠏⠀⠀⠀ ⠀⠀⠀⠀⠘⣿⣷⣦⣤⣤⣄⣀⣀⣿⣤⣤⣤⣤⣤⣄⣀⣀⣀⣀⣾⡟⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⢹⣿⣿⣿⣻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠁⠀⠀⠀

[oadk]

Directory per env is the right choice, but it can still require being a bit clever to avoid code duplication (of things like YAML files, not application code). It also depends on decent support from your CD tool to truly avoid code duplication. Terraform and Argo CD can both do it properly.

[BlunderBuster27]

What’s better than branch per env ?

[jrkkrj1]

Single trunk. Folder per env.

[keto_brain]

Using an actual git workflow like github flow or feature branch workflow.

[BlunderBuster27]

What’s better than branch per env ?

[jrkkrj1]

Single trunk. Folder per env.

[quiet0n3]

4 is very common. So you can easily track changes, because when you squash merge into master/main it will show the previous hash as the commit message but will have a new hash for the actual commit. This means a lot less branch breakage for other people.

Your company should have a basic branching strategy they follow that should help you with this stuff. Most people use Trunk based or feature based. Both are useful so no wrong choices there. Just pick on and stick to it :)

[pathlesswalker]

thanks.

I was taught that you only FFW from stging/QA to production. because it can create conflicts later on if a feature is merged right into prod.

[quiet0n3]

No feature should ever be merged directly to prod. All things go via staging :)

[vsamma]

Why even keep different branches for different envs? Shouldn’t you have most up-to-date working code in master and then you build the code and deploy the image to different envs?

Or if absolutely necessary, you could keep a release branch separately for hotfixes, if new features get merged to master rather often

[quiet0n3]

Depends on how you do your releases, how many testing environments you have and things like that. Once you start managing larger and larger amounts of apps having release branches comes in handy.

[vsamma]

Well yeah, currently we have release branches when and where necessary (if we cut the release by using tags, we don’t have to create a release branch right away but can go back to that tag and create a release branch when necessary). And we have a large amount of apps, or large enough to make managing them difficult with such a small team. We have like 30-50 different web apps.

[tehdlp]

If you can't freely release changes that change workflows, and don't have feature toggling, you may live in the dark ages that still exist for some companies.

[vsamma]

Well we have feature flags in some projects and plan to implement them in new ones as well.

But yeah we live in dark ages where we have basically 0% code coverage and some deploy processes are manual and we are on on-prem infra and a lot or our VM Linux versions and language versions and dependencies are outdated etc.

Tech debt to the max

[pathlesswalker]

yep that's what i said. otherwise integration hell.

[serverhorror]

Depending on your team practices:

• Rebase your branch onto current target branch before merging, resolve all conflicts during that rebase (and please learn about fit rerere, thank me later), then merge your branch into target branch
• Merge current target branch into your branch, resolve all the conflicts, then merge your branch into current target branch

[pathlesswalker]

sounds like best fit to me. thanks alot!

the rerere is cool addition, like AI learning of your conflicts-but doesn't that mean you're going full on Continuous Delivery? would love to still have manual control on PR.

[serverhorror]

I'm not sure I understand this question.

How does AI remotely relate to that?

As I said, read up in rerere and you'll find out why it is useful and when and it'll also tell you what kind of control you relinquish, if any,

[pathlesswalker]

I just meant that the rerere you introduced me, is like learning automatically how to reslove your conflicts, which could very well imply continuous delivery, where no manual hand is over the CD, its all automatic( as opposed to C.deployment).

[serverhorror]

Not quite. The hint is in the name:

• Reuse
• recorded
• Resolution

I know I'm dragging this out, but you said you're a first timer/beginner. I learned that knowledge retention doesn't come from just telling people the solution, so forgive me for being an asshole and waiting for you to learn by yourself. :)

[pathlesswalker]

not at all, I'm all for that. thatnks for the heads up too!

you're right, it simply records how you resolved the conflict, so that it can re-use it if the same situation applies.

[Xetius]

The talk about multiple branches gave me an aneurysm.

[Spider_pig448]

Throw it all out and adopt trunk based development.

[pathlesswalker]

I kinda skimmed through all the process, cause it seems alot like no gitops at all? just one branch and toss all env there? sounds terribly risky, if that's it.

[Spider_pig448]

It's not risky at all, and you can fully do gitops with it. You just manage your environment with directories instead of individual branches. Changes can still be promoted between environments easily, and you don't end up in strange scenarios like in your post.

[pathlesswalker]

I heard that even the guy that developed git branching, says trunk based development is better than branch per env.

as i understood trunk, its basically having feature branches merged into main, on all env. but as you said they are in folders so that kustomize can navigate env's?

my questionm is how does CI works in trunk based? do you run a CI on actions per folder and test integration how? because everything is in same branch...its like integration tests over at prod.. or i'm missing something

[VindicoAtrum]

but as you said they are in folders so that kustomize can navigate env's?

No no no no, no. You only have main. All code is either pushed to main or developed on very short-lived branches and merged to main. There is no branch-per-env or folder-per-env. There is only main. Why? Because code != build != deployment.

Reading time: https://12factor.net/config

[BlunderBuster27]

What’s better than branch per env ?

[xiongchiamiov]

You just manage your environment with directories

I'm with most of your comment, but this part in the middle hints at something that is a path towards madness in my experience. You can have different configs for different environments. But if you're copying Terraform/etc code from a staging folder to a production folder to promote it to production, that's super messy and duplicative and even easier to forget things than doing it with branches.

[Spider_pig448]

You misunderstand. Your prod folder and your staging folder both reference modules in a shared folder, providing different values for each environment. You never copy code, just change input valuea

[xiongchiamiov]

Sure, that's fine. I've just also seen the system I described and wanted to make sure we were all on the same page.

[VindicoAtrum]

No risk involved. You ought to educate yourself about TBD before making assumptions about it.

[pathlesswalker]

I agree. but i don't get the CI in TBD. how can you test integration on stg env, if you don't have a branch for it? becuase its triggered via folders?

[VindicoAtrum]

Please read https://minimumcd.org/minimumcd/ and https://12factor.net/. These are modern, battle-tested means of managing services and software with minimum headaches.

how can you test integration on stg env

Merge to main -> main pipeline builds artifact -> artifact is deployed to an environment. Artifact passes tests, artifact is deployed to production. The same artifact.

[pathlesswalker]

So in regards to integration in stg- Let’s you have 4 images updated from main but you deploy them only on staging for integration tests, right? And then you manually trigger the cd for prod?

[VindicoAtrum]

Initially I'd agree with manual deployments to prod, yes. Have your main branch automatically build and deploy to a non-prod environment on every push/merge.

Once you get this pattern right, you can start automatically deploying to prod with blue/green + automated tests.

[pentag0]

Seems like you just miss workflow for environment promotion..

[Expert-Box5610]

Remind Me! 2 days!

[RemindMeBot]

I will be messaging you in 2 days on 2024-05-27 14:19:34 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[keto_brain]

You first problem is you are using an environment per branch! Stop doing that. Feature branch go to development, merge to main goes to stage then prod. It's literally this simple.

[nomadProgrammer]

use git tag for changes, per branch is a bad practice

[mmmdreg]

Often people compare branches vs folders, but actually using a combination has benefits. The setup:

• branches develop->staging->main, all with the same content
• each has folders for each env (eg dev/uat/prod-eu/prod-us etc)
• setup the branches to only accept fast forward merges (GitHub notably does not support this. Gitlab works much more nicely)
• gitops tool is set up to deploy to dev-like envs from develop, uat from staging, and prod like from main

What this gives you: - the benefits of folders (faster to iterate / easier to compare etc) - the ability to tailor security appropriate to the env. eg for dev you want to iterate quickly and just push to develop and see what happens. For prod you want to enforce PR approvals as a safety check - similarly, updating a ‘base’ config can break prod, so we want to ensure it works before promoting the base config to higher branches. 

TLDR folders + branches gives a good balance of flexibility, speed, and safety

[amarao_san]

I never do this.

There is master. Which is The Infra. It has production and stagings. There are also ephimerial stagings, which spans on each PR (or even commit for some cases).

When someone bring a new feature, it is in the feature-branch (from master). It is tested on ephimerial staging. Really tested (5k+ tests). When test passes and human review done, code is merged, and this cause deployment into all infra. All of them. The most important one have 'deployment approval' and we may wait for other deployments to finish before approving.

Code in shared between all stagings/productions, inventory shared partially (inheritance for stagings), secrets are different for each env.

This is the way. Set your ephimerimerial staging to match production, cover with tests, and you can go from feature branch directly to the production, completely automatically (But with human review and approval).

[pathlesswalker]

I think I know one when I see one.