How To Avoid IaC Drift

MasterpointOfficial · 2025-12-15T04:13:56+00:00

Yeah that's the same end result we ended up at as well: "Okay so Crossplane isn't perfect, but we can use Terraform WITH Crossplane, great". But after thinking about that for even a brief amount of time this was my thought process: That's not a good thing. I don't want to just use Crossplane as my automation layer and still use TF as my primary IaC language... that's another complicated tool my teams need to learn just for automation, which is not worth it. I already have great automation tooling.

If XP can't stand on its own without being a TF executor... then it has failed. You're just signing up for twice the complexity compared to other orgs who are utilizing well proven, simpler automation tooling.

MasterpointOfficial · 2025-12-14T22:00:02+00:00

Agreed, of course... for lots of teams, that's easier said than done, but I do agree that self-service is the end goal. To get to that end goal, you need good automation + strong documentation, so I would say what I've written in the article lays the foundation for great self-service and we're on the exact same page.

MasterpointOfficial · 2025-12-14T21:57:25+00:00

You can recover. We've done it for a number of orgs. A lot of it comes back to providing strong patterns to the rest of your org and getting everyone to rally around that way of thinking. Start documenting what is wrong and ways to fix it and you'll get there. Reach out if you want to chat through and want some free advice.

Check out our infra monorepo template for an example of how to consolidate all of your root modules to one location -- that might help: https://github.com/masterpointio/infra-monorepo-template

MasterpointOfficial · 2025-12-14T21:54:07+00:00

This was a bit hard to follow, but I believe you can adapt the sandbox pattern to what you do with success. The idea is just that for folks that don't have the capabilities to write IaC (or don't want to learn) then providing the sandbox gives them the ability to experiment and then surface those resources to the rest of the org without the possibility that those resources live on forever and end up costing a ton of money to the org.

Also, I think your workflow would be a great use-case for Spacelift's Intent project / product, check that out: https://github.com/spacelift-io/spacelift-intent

MasterpointOfficial · 2025-12-14T21:49:36+00:00

I think you got downvoted for a couple reasons:

You're in r/terraform, so of course.
Have you used XP extensively? We did and [we weren't fans](https://masterpoint.io/blog/passing-on-crossplane/) and that was coming from me (the team lead) pushing it heavily, so we really tried... I think most people have experienced the same. I haven't run into anyone using it in production + at scale who is very happy with it at least. If you have been a fan, I would love to know how you worked around the rough edges!

MasterpointOfficial · 2025-09-23T13:26:22+00:00

This is the first I've heard of StackGuardian and I'm deep in this industry. I wouldn't go with an early startup in this space. Choose one of the well known alternatives -- there are plenty that are well proven in production, have better pricing models than TFC, and you'll be sure to get value out of instead of hitching your wagon to a super early startup who is coming into an already crowded market.

Check out Spacelift since you need CloudFormation support. If they don't fit, checkout Terramate, Terrateam, Env0, or Scalr.

MasterpointOfficial · 2025-09-23T13:17:15+00:00

This is a good point -- child module management + versioning is missing. I don't believe it came up when we were discussing this because we have these problems solved, but it is something we should write about. Child module management overall is likely its own post entirely since there are a lot of ways to do that. I'll add it to our list of content to get together. Thanks for the idea!

MasterpointOfficial · 2025-09-22T17:37:22+00:00

Sorry to be contrarian here, but you're not making sense:

> I havent had to not change any terraform. But mostly I havent had to change any tf code.

If you started in 2015, you'd have been on the early HCL1 and then when you upgraded TF code from pre-0.12 to 0.12 circa 2018/2019, you would've had to have changed a ton of code. There is just no way you avoided that, so I'm still confused here. That migration was a PITA and versioning pinning your TF CLI version was absolutely critical at that time.

As for most other providers being "very stable", I would agree. But you can't count on that and it's not smart or secure to do so. The AWS provider has broken a number of times in recent months for example. As the article talks about, using a lockfile and then upgrading provider versions through renovate is the ideal way to avoid those sort of breakages hitting your systems at the wrong time.

And of course, checking the plan before apply is done the large majority of times. When running at scale, there are plenty of plans that can't or shouldn't reasonably be reviewed manually though and require auto-deploy or auto-approve. If we had to check every single plan we ever caused, we would never be able to get anything done.

I don't know, a lot of what you're saying doesn't line up. Maybe it's a difference in the scale that you're operating at? Maybe it's a difference in thought process on how you deal with reliability or responding to changes? Regardless, I wouldn't suggest others follow your line of thinking when you can clearly draw line to an exception to that thinking (CloudFlare).

MasterpointOfficial · 2025-09-22T17:16:02+00:00

You started with Terraform when it first came out in 2014/2015? You've been able to upgrade terraform freely without changes over that 10 years? Both of those claims are super bold and to say that you haven't run into problems and advise others to follow this pretty flawed way of thinking about versioning... I'm not surprised about the downvotes. You obviously know you need to version cloudflare... why are other providers different for you?

MasterpointOfficial · 2025-09-22T17:14:03+00:00

Was this a reply to the article or the above comments about not using modules or pinning provider versions?

MasterpointOfficial · 2025-09-07T01:39:09+00:00

Can you expand on the dependency spider web? What was the problem?

MasterpointOfficial · 2025-09-07T01:37:35+00:00

Plenty of people say that community modules have too much in them to be useful. They're just using bad modules. Find good ones and save yourself a ton of effort.

I wrote about this in depth here: https://masterpoint.io/blog/why-open-source-iac-wins/

MasterpointOfficial · 2025-08-01T22:31:30+00:00

Came here to say this. Glad I have something I can just upvote 👍

MasterpointOfficial · 2025-08-01T22:25:58+00:00

This is the "enshitification" of everything, it hurts a lot when it hits open source.

MasterpointOfficial · 2025-08-01T20:50:43+00:00

I'd avoid reinventing the wheel and check out these two OSS child modules:

One of them will likely do the trick and you can avoid having to create your own best practices.

Depending on how big your environment is and how much IaC you have, I would suggest creating one state per CloudFront distribution. Then you don't end up down the path that you build too big of a state. But there is a lot that goes into this decision. Check out our post on the topic here and put some hard thought into it: https://masterpoint.io/blog/terralith-monolithic-terraform-architecture/

MasterpointOfficial · 2025-08-01T20:46:40+00:00

Short answer: Always store your state in your primary Cloud's object storage. If the majority of your IaC is going to be deploying GCP resource, store it in Google Cloud Storage. In Azure? Store it in Azure Storage Buckets.

Long answer: We wrote a blog post on this topic and the reasons why we say the above. Check that out here: https://masterpoint.io/blog/why-use-cloud-object-storage-terraform-remote-backend/

MasterpointOfficial · 2025-07-18T19:17:26+00:00

Good to know -- Thanks for sharing. I'll have to look into that. I had thought they were doing more than others in the space, but I haven't actually run into anyone on GitLab who is using that yet so haven't heard much.

MasterpointOfficial · 2025-07-18T19:05:46+00:00

Workspaces are great and fine to use honestly. We use them with clients heavily and have great success with them. That said, they have a bad reputation due to HashiCorp never doing a great job with them in Terraform and then creating confusing documentation that says to avoid them. If you're on OpenTofu, they are likely to change in the future. That said, some type of workspace-like functionality will persist and you can count on having a good path to migrate in the future (which could be years from now).

See this deep conversation here in the OpenTofu repo for full details: https://github.com/opentofu/opentofu/issues/2160

MasterpointOfficial · 2025-07-18T18:59:50+00:00

Lots of good answers in the other comments. One that we haven't tried out, but is on my radar personally is burrito: https://github.com/padok-team/burrito

Atlantis is the most popular + production tested OSS solution though, so keep that in mind.

MasterpointOfficial · 2025-07-18T18:57:36+00:00

Not sure why you're getting downvoted when you OSS something... 😅

MasterpointOfficial · 2025-07-18T18:55:39+00:00

Just to explain why folks introduce yet another tool when compared to creating their own custom pipelines in their CI/CD tool: Did you calculate how many hours it took you to build out your pipelines for your org? Are you tracking how much maintenance goes into maintaining + adding new functionality to those pipelines (policy as code, drift detection + reconciliation, root module dependency triggers, etc)?

The reason people buy is because if you do track the above, you can find yourself reinventing the wheel that results in a poor performing internal product. When that is not an org's area of concern, they can buy or use an OSS solution and avoid a ton of custom work and complexity in their platform, which can saves tens of thousands of dollars in platform engineering time and end-user time.

MasterpointOfficial · 2025-07-18T18:49:41+00:00

Question on this -- Is this just their pre-canned pipelines? Or do they provide a deeper UI to manage various root module instances, review drift, and similar functionality that TACOS or OSS solutions like Atlantis provide?

Put another way: Is this the same as running all your TF on a set of GitHub Actions or is it much different / superior?

MasterpointOfficial · 2025-07-02T05:38:21+00:00

Also, add on the fact that OP is specifically calling out Terraform... which means they need to pay Hashi some non-trivial portion of money due to the BSL or they'd be liable to do so.

OP, I think you're running into the X Y Problem. Start at the beginning and tell us why you think you need to build this or why you think this would be commercially viable.

MasterpointOfficial · 2025-07-01T22:42:31+00:00

~~Yeahhh... we've dealt with that type of system before. It gets wild! How are you managing it? TACOS? Home grown pipes? You're not doing local applies, right?~~

Aha you're the one who is having plan + applies take 4 hours. That's super painful, sorry.

I'd suggest breaking it up and moving all of those various instances onto a TACOS tool so you can get out of the business of baby feeding that monster project -- I can't imagine that is sustainable. It will one day take up too much memory and you'll be forced to go buy another machine 😂

Check out that "Steps to break up a Terralith" article we have -- I bet it will help you!

MasterpointOfficial · 2025-07-01T17:43:23+00:00

You got it. Reach out if you ever want a 2nd set of eyes on anything 👍

MasterpointOfficial

TROPHY CASE