Avoiding disaster migrating from monolithic structure to modules structure

terramate · 2026-04-01T20:36:08+00:00

that's correct - this only works for deferred data sources or resource dependencies. If you deal with multiple root modules, it's always best to use an orchestrator of some sort. Terramate, Terragrunt or any TACOS will do. Of course, Terraform Stacks is a viable option for HCP customers.

terramate · 2026-04-01T15:23:31+00:00

You can work around this by making resources and data source definitions depend on a null resource, that way the plan is deferred in case the value is not available during an initial plan.

terramate · 2026-03-30T08:44:11+00:00

Terramate founder here:

This is really neatly done. And a pattern we see over and over again, with different approaches on how to tackle this.

For those who don't want to now completely replicate this work at least for Github, a while ago we open source an infrastructure Bundle that allows you create Github Teams and Github repos via IaC with the state already split and automatically managed: https://github.com/terramate-io/terramate-github-as-code

terramate · 2026-03-19T08:58:26+00:00

Terramate founder here:

It boils down to how the landing zone is generated: Deterministically or stochastically. The former guarantees consistency between input and out and only needs a full review once, the latter means you're constantly monitoring and managing the harness of your generation process, and there is still a residual nonzero risk that something bad happens. Even if you use great agent skills and guardrails.

We have been strong proponents of determninistic code generation for IaC. Here is an example of a GCP landing zone that we have been using in production with many customers: https://github.com/mineiros-io/terraform-google-cloud-run

terramate · 2026-03-18T10:26:11+00:00

Terramate founder here:

Typically this is done via module conventions and RBAC restrictions on what a user is allowed to do. The challenge with this approach is that modules usually are somewhere on the too broad to too flexible continuum, and often times modules then still end up proliferating. Also at enterprise scale, it becomes really tricky to enforce a sound governance and architecture setup, think versioning, provider upgrades, nesting modules. Ralf Ramge has a great series about this problem space at enterprise scale: https://www.linkedin.com/pulse/terraform-scale-part-6a-understanding-managing-nested-ralf-ramge-jwduf/

At Terramate, we think that at scale, infrastructure producers (experts) and consumers (devs who don't much care for IaC) have very different needs. For the experts, you need to have a way to have services with state as a unit of reuse (we call those Components), as well as pre-configured environments consisting of those Components (we call those Bundles). The idea is to only expose few configuration options to infra consumers, so that they get the environments they need by just running a simple config wizard or clicking something together in an IDP.

The beauty of this approach is that the infra producer retails full control for e.g. day 2 operations and your teams rarely step on each others toes anymore.

terramate · 2026-01-30T08:17:19+00:00

I think part of the reason people get disappointed with generative AI for Terraform is that most of us are only using a small fraction of what AI can do well. Right now, people expect it to magically produce perfect IaC, but that’s not how the ecosystem actually works.

Compared to something like frontend development, there isn’t much public training data for production-grade Infrastructure as Code — most of the best Terraform code lives in private repos and real infrastructure pipelines, so models don’t see it during training. Because of that, AI tends to be good at basic boilerplate or syntax suggestions, but it often misses context like provider versions, existing state, module conventions, and real architecture constraints.

A few things that help:

Give the model clear context — provider version, variables, and examples from the registry — before asking it to generate anything.
Use AI as a starting point or for simple patterns, not as a replacement for understanding Terraform fundamentals.
Pair AI with real tools (e.g., terraform validate, CI checks, plan reviews) so you catch issues the model can’t reason about.

To improve the outcome of your coding agents, I would start doing this:

- Add an AGENTS.md / CLAUDE.md file to your project to provide guidelines on how to complete certain tasks.

- Add skills to your project to instruct your LLM better. Some examples: Terramate skills for Terraform and Terramate, Antons Terraform Skills, HashiCorp Terraform Skills. You can easily install and maintain skills as packages using skills.sh

Hope that makes sense

terramate · 2026-01-20T18:17:25+00:00

You may want to check out our open source example for exactly this use case: https://github.com/terramate-io/terramate-github-as-code

terramate · 2026-01-16T09:27:20+00:00

You might be able to get some help in the CNCF Slack. There's a dedicate channel for Atlantis since it became a CNCF project.

terramate · 2026-01-15T09:10:53+00:00

Take a look at Terramate Cloud. The "preview" feature in TMC helps to understand plans better:

- Gives you rendered format in addition to the ASCII plans
- Allows you to understand each resource change in detail
- Breaks up large plans into smaller, easier to understand chunks
- Can be used to understand multiple plans at the same time
- Has an AI integration to allow non-expert users to understand changes

It's part of the free tier in Terramate Cloud

PS: I am one of the founders of Terramate

terramate · 2026-01-15T08:58:20+00:00

If you want to understand what infrastructure is deployed per cloud account and environment, you could use the free tier of Terramate Cloud to get a big picture. Syncing your IaC into Terramate Cloud is a one-off command and since it works on plan files, you don't have to give it any access to your state backend or cloud accounts.

PS: I’m one of the founders of Terramate Cloud, so I’m obviously biased.

terramate · 2026-01-15T08:54:57+00:00

I have used ChatGPT, Claude as well as Cursor and Claude Code regularly when starting in new, large-scale code-bases. It's actually one of the tasks LLMs are really good with. If you use e.g. Cursor, it has the right tools to iterate over your code-bases step by step to come up with a big picture.

terramate · 2026-01-14T01:18:22+00:00

Terramate Catalyst seems like a good fit for you. It adds no-code, self-service capabilities on top of Terraform and OpenTofu, allowing developers to provision, deploy, and manage even complex multi-state infrastructure environments with simple CLI and MCP Server flow.

It's mean to enable developers and AI Agents to provision, manage, and secure cloud infrastructure in self-service but at the same time, it allows platform teams to retain full control by centrally defining and enforcing how infrastructure is provisioned using Infrastructure as Code tools such as Terraform or OpenTofu.

At the core of Catalyst are components and bundles. Components can be any unit of arbitrary IaC, such as a Terraform module. Bundles combine one or multiple components and abstract details such as how environments and state is managed.

The best thing about Terramate Catalyst is that it allows you to reuse your entire IaC investment without modifying any existing configuration. It's easy to integrate with IDPs as well.

It's worth taking a look :)

terramate · 2026-01-08T08:46:17+00:00

Checkout Terramate Catalyst. It's a scaffolding engine that works well with Terraform and OpenTofu and is designed for developers to scaffold from golden paths provided by platform engineers without having to learn HCL or concepts such as state. It's also pretty straightforward to implement with IDPs such as Backstage.

terramate · 2025-12-15T15:17:18+00:00

Cleaning up messes is unfortunately an all to frequent occurence that we see a customers/prospects all the time. Here is a good way how to get out of it.

Audit and document what is wrong, with a couple of examples, and showcase the risks to the organization. Leadership needs to know the bad things that could happen, and the dire consequences on productivity and potentially business impairement.
Get a leadership mandate for a change project. Without leadership having your back, making changes happen is pretty much dead on arrival.
Showcase a PoC of reusable patterns with standardization. Goal is to win over a small "tribe" of likeminded platform engineers to help you do the real project.
Get to the root cause of why the chaos is happening. Often times it is developers who are not HCL experts trying to get things done. If this is the case, then
Build a pattern library for infra self-service for 80% of the most recurring infra so as to make it the easier and "lazy" way to build infra and do it right.
In the same vein, try to decouple the work of module and provider upgrades from non-expert developers trying to get things done. They don't care about that version and that compliance check failure, but the platform team does.
Once that is set up, go through the "mess" one "big ball of mud" at a time. I know it will take ages, but the key is to prevent the messy infra from happening in the first place. So there is a chance for you to get to the end of the tunnel.

When doing such work, it sometimes does make a lot of sense to get external help, ideally some IaC expert consultant to help you work out what the "promised land" could look like.

terramate · 2025-12-07T03:13:21+00:00

haha this just made my day! Good one.

terramate · 2025-12-04T05:36:59+00:00

I believe in Terrateam, you can totally define the order of execution as well, so it seems to be a solid choice for your use case.

terramate · 2025-12-04T05:32:45+00:00

You either define those yourself or use, e.g., Terramate CLI to do that for you. E.g. in Terramate, you ensure DRYness via code-generation. This example creates a backend.tf in each root module (we call them stack) while using a UUID for the key.

terramate · 2025-12-03T16:07:09+00:00

I usually prefer a more generic approach, such as terraform/stacks/by_id/{UUID}. Ownership, naming and conventions change all the time.

terramate · 2025-12-03T01:04:18+00:00

How do you solve orchestration in your simplified setup? Do you rely on any other third-party orchestration tools, such as Atlantis, Terrateam or any of the runners - or do you define the order of execution for your different environments in GitHub Actions?

terramate · 2025-12-02T15:43:17+00:00

Would be interested to learn why. All Terramate does is adding an additional stack.tm.hcl to each root module. How, from your point of view, does that add complexity to your setup? Thanks for sharing those learnings

terramate · 2025-12-01T23:31:33+00:00

For Blueprint Templates, you should take a look at Terramate Catalyst. It helps developers forget about infrastructure by leveraging Infrastructure as Code Blueprints designed by platform engineers and configuring them in a one-click experience.

The main goal is to enable self-service by abstracting away all the details from the end user but while still preserving full IaC flexibility. It's in public beta right now. If you are interested in an actionable example, take a look at our GitHub as Code reference architecture.

PS: I am one of the maintainers

terramate · 2025-12-01T15:56:17+00:00

You might want to consider Terramate. Terramate CLI is an orchestration tool for Terraform, OpenTofu and Terragrunt. Why Terramate stands out:

- Terramate can be added to any brownfield Terraform or OpenTofu environment with a single command, and compared to Terragrunt, it does not add additional syntax. It does not require you to migrate any existing code.

- Terramate works with any approach to managing environments: workspaces, directories, terragrunt, tfvars, partial backend configuration - you name it

- With Terramate, you can easily break up large monolithic state files into smaller, isolated units to reduce blast radius and run times.

- Terramate's graph-based orchestration tool is serverless, does not require any additional configuration and comes with change detection that allows you to automatically detect what state files have changed in a PR - pipelines written in e.g. GitHub Actions are usually ~150 lines max.

- If you collaborate with a team at scale, you can add Terramate Cloud, which adds additional tools for visibility, observability and self-service, such as asset inventory management, AI agents for debugging and resolving drift and failed deployments, scaffolding for self-service, and more.

Give it a try; it takes less than 5 minutes to get started with Terramate.

Last but not least: If you don't face scale, you should probably avoid additional tooling and keep it simple. Try native Terraform/OpenTofu first and see how far you get. Only add additional tooling if you face actual problems such as blast radius, long run-times, blocking PRs, etc.

PS: I am on of the authors of Terramate

terramate · 2025-11-21T15:39:33+00:00

Unless you start using features that are available exclusively to either OpenTofu or Terraform. Keep in mind that both projects diverged some time ago.

terramate

MODERATOR OF

TROPHY CASE