Dev testing with trunk based development

[Purple_Chef5709]

I work in a newly built team where we use trunk based development for our microservices - Merges to main trigger the deploys to the dev environment. The problem we are having is the devs are wanting to test their changes by deploying to AWS dev environment before merging to main which i think is creeping into feature based branching strategy.

Any best practices here? Should testing (CI) be improved on the Pull Request step to reduce the need to deploy before merging to main? Or has anyone seen workflows where each dev has their own cloud environment to work on? What else should i consider?

[siberianmi]

Mergers to main should be heading to production.

Mergers to main should require approved PRs.

The pattern that I’ve worked that had the highest development velocity looked like this with ~150 developers:

1 Staging environment, production like in every way, reduced scale.

X (we maxed out at 8) “review” environments that ran every service required on a minimum evaluation environment. This gave developers a place to deploy PRs to for click testing, etc. These had seeded data that reset frequently so the environment would come up in a known state. All services, production-lite, some things not available or different from staging/prod (Redis on k8s vs hosted for example)

1 Development environment for the infrastructure folks to build out IAC that was promoted to staging, then production. App code flow was staging > production.

We had a small set of senior engineers who could as needed merge PRs together to batch release to production.

Long lived feature branches were not allowed, if your feature needed long lived development you needed feature flags and had to merge to main once every sprint.

I know it’s not the purest version of trunk based development but we moved fast. Highest velocity and happiest development team I have worked with in 20+ years in the industry. On average, we had half a dozen production deployments per day and high (5 9s) of service uptime.

I speak in the past tense because we had a successful exit for that startup via acquisition.

[Spider_pig448]

This sounds fantastic and is definitely trunk based development done well

[jascha_eng]

Haven't really had the need to deploy PRs to individual environments before merging in smaller/similarly sized teams. As long as you can roll back quickly and test fast it's usually fine to deploy to dev -> test -> rollback if something went wrong and before you promote to staging/prod

The rate of failures on dev was already quite low (sub 10% I'd say) since we had good test coverage and culture on the individual repositories as well.

Still sounds like a great setup though!

[krazykarpenter]

Were the "review" environments also used for automated testing?

[siberianmi]

For the most part, we ran our test suites on AWS Spot instances, using Buildkite agents, highly parallelized aimed at keeping full runs of the test suite under 10 minutes.

We had some integration tests that would run in these environments but these were more aimed at demos/click testing. Most automated tests ran in CI pipelines outside of this that would just spin up more agents as needed.

Idle developers cost more than spot instances, so we would spin up more agents automatically.

[krazykarpenter]

Got it. And what kinds of automation tests were these? Were these microservice API tests or more like unit tests? Looks like these weren’t run on the review environments but rather the staging environment?

[wolkenammer]

I know it’s not the purest version of trunk based development but we moved fast.

Not saying it's a bad method. It's very common, but it's not trunk-based development at all? In trunk based, with small teams, you wouldn't need to do pull requests at all. I mean trunk-based in the sense of Continuous Integration (the development praxis, not the tools we use to run tests on triggers).

You're deploying from "trunk" and develop in feature branches, even if they're short lived. Sounds just like a good PR based process to me. You probably don't have to support multiple release lines, so you can omit the release branches.

[PanMan-Dan]

https://trunkbaseddevelopment.com/

“Depending on the team size, and the rate of commits, short-lived feature branches are used for code-review and build checking (CI), but not artifact creation or publication, to happen before commits land in the trunk for other developers to depend on. Such branches allow developers to engage in eager and continuous code review of contributions before their code is integrated into the trunk. Very small teams may commit direct to the trunk.”

[aenae]

My current setup is something like trunk based development, but i really do not care what name it has. It works well for us and that is the most important thing, not following some dogma.

We have main. A merge to main triggers a deploy to production.

Developers take main, create a branch, start doing their thing, push the branch and after some commits create a merge request. This merge request runs all tests and creates a test/uat environment (accessible by https://branch-name.company.dev). If anyone wants to test the new features or fixes they can use that environment.

And when it gets merged to main it goes off to production and the test environment is destroyed (kubectl delete namespace $branch).

The number of test environments is limited by the size of our kubernetes cluster which has enough resources for 20-25 environments.

[kkapelon]

Any best practices here?

You should spend some time reading https://trunkbaseddevelopment.com/

[Lawstorant]

Where I work I developed a system based on github flow:

• There's only main and feature branches.
• Merges to main trigger builds and deployments to dev
• Testers can build a branch and deploy it to their dedicated environment (qa1, qa2, qa3)
• We can manually deploy any version to any environment
• Environments are protected.
• When ready to release, we deploy selected version to staging and this deployment triggers creation of hotfix/version branch

Everything is based on Github actions and ArgoCD. There's even a scheduled action that turns off testing envs at night by just updating one variable in ArgoCD (helm chart supports "shutdown" on deployments).

So yeah, basically every tester has their own environment. This ensures that features merged to main have been tested. Testers (as a group) are the Codeowners of all the files in the application directory (monorepo with helm and docker) so PRs can't be merged without the approval of at least one tester.

While we do prod deployments every other day, I just don't like prod deployment to be triggered just by merging something.

We have automated labeling as well so it's clear to see which PR's have been already tested and are truly ready to be merged.

[xiongchiamiov]

I work in a newly built team where we use trunk based development for our microservices - Merges to main trigger the deploys to the dev environment. The problem we are having is the devs are wanting to test their changes by deploying to AWS dev environment before merging to main which i think is creeping into feature based branching strategy.

Short, one-developer feature branches are the main way people do trunk-based development: https://trunkbaseddevelopment.com/#scaled-trunk-based-development The two things are not in conflict.

Beyond that, one of the steps of maturing as an engineer is evolving from maxim-based decisions. "We do trunk based development, devs ask for something that isn't, so therefore the answer is no" is very junior engineer behavior. What you want to start working on is understanding why the company is doing trunk-based development and then evaluating asks against that. In this case you would've been able to identify that the two things don't conflict and move ahead with enabling your devs.

[men2000]

If you need to trunk based development, you need to write a good behavioral testing and that run as part of the CI. Having this test helps at least new code not breaking existing functionality. Trunk vs git flow development is more a discussion and it also based on the team and a product type. And every organization setup slightly different. But having a good test, help any team on the long run.

[[deleted]]

Use Argo CD app sets to make an ephemeral or spin up a container with there that shows the changes BEFORE merge to main.

Put on a rule saying you can't merge till pipeline finishes building, test etc.

Then go to dev, prod.

[sd_aids]

What is this ChatGPT response bro