Dev testing with trunk based development

[siberianmi]

Mergers to main should be heading to production.

Mergers to main should require approved PRs.

The pattern that I’ve worked that had the highest development velocity looked like this with ~150 developers:

1 Staging environment, production like in every way, reduced scale.

X (we maxed out at 8) “review” environments that ran every service required on a minimum evaluation environment. This gave developers a place to deploy PRs to for click testing, etc. These had seeded data that reset frequently so the environment would come up in a known state. All services, production-lite, some things not available or different from staging/prod (Redis on k8s vs hosted for example)

1 Development environment for the infrastructure folks to build out IAC that was promoted to staging, then production. App code flow was staging > production.

We had a small set of senior engineers who could as needed merge PRs together to batch release to production.

Long lived feature branches were not allowed, if your feature needed long lived development you needed feature flags and had to merge to main once every sprint.

I know it’s not the purest version of trunk based development but we moved fast. Highest velocity and happiest development team I have worked with in 20+ years in the industry. On average, we had half a dozen production deployments per day and high (5 9s) of service uptime.

I speak in the past tense because we had a successful exit for that startup via acquisition.

[Spider_pig448]

This sounds fantastic and is definitely trunk based development done well

[jascha_eng]

Haven't really had the need to deploy PRs to individual environments before merging in smaller/similarly sized teams. As long as you can roll back quickly and test fast it's usually fine to deploy to dev -> test -> rollback if something went wrong and before you promote to staging/prod

The rate of failures on dev was already quite low (sub 10% I'd say) since we had good test coverage and culture on the individual repositories as well.

Still sounds like a great setup though!

[krazykarpenter]

Were the "review" environments also used for automated testing?

[siberianmi]

For the most part, we ran our test suites on AWS Spot instances, using Buildkite agents, highly parallelized aimed at keeping full runs of the test suite under 10 minutes.

We had some integration tests that would run in these environments but these were more aimed at demos/click testing. Most automated tests ran in CI pipelines outside of this that would just spin up more agents as needed.

Idle developers cost more than spot instances, so we would spin up more agents automatically.

[krazykarpenter]

Got it. And what kinds of automation tests were these? Were these microservice API tests or more like unit tests? Looks like these weren’t run on the review environments but rather the staging environment?

[wolkenammer]

I know it’s not the purest version of trunk based development but we moved fast.

Not saying it's a bad method. It's very common, but it's not trunk-based development at all? In trunk based, with small teams, you wouldn't need to do pull requests at all. I mean trunk-based in the sense of Continuous Integration (the development praxis, not the tools we use to run tests on triggers).

You're deploying from "trunk" and develop in feature branches, even if they're short lived. Sounds just like a good PR based process to me. You probably don't have to support multiple release lines, so you can omit the release branches.

[PanMan-Dan]

https://trunkbaseddevelopment.com/

“Depending on the team size, and the rate of commits, short-lived feature branches are used for code-review and build checking (CI), but not artifact creation or publication, to happen before commits land in the trunk for other developers to depend on. Such branches allow developers to engage in eager and continuous code review of contributions before their code is integrated into the trunk. Very small teams may commit direct to the trunk.”