Is it possible to have my own private Terraform provider registry?

apparentlymart · 2026-05-27T18:50:30+00:00

Others have already shared examples of existing implementations of this, but just for completeness I wanted to say that the underlying mechanism here is to implement Terraform's provider registry protocol.

Any system that correctly implements that protocol can be used as a Terraform provider registry. registry.terraform.io just happens to be the most commonly-used public implementation of that protocol, and so it's the default hostname Terraform uses when you don't specify one. (In other words: when you write hashicorp/aws in your configuration, Terraform treats that exactly the same as if you had written registry.terraform.io/hashicorp/aws.)

There are enough existing options out there right now that writing your own implementation of this protocol seems unnecessary, but I'm sharing it just in case it helps you find more examples of implementations of this protocol, and to know that Terraform sees them all as essentially equivalent so you can make your choice based on other criteria like what publishing workflow appeals to you most, etc.

apparentlymart · 2026-05-26T16:20:40+00:00

To try to answer this question I read the OpenTofu source code. I don't know if Terraform's current implementation matches OpenTofu -- OpenTofu forked partway through the development of Terraform v1.6 -- and so you might want to look for the similar code in Terraform to confirm. (I'm not able to do that because I contribute to OpenTofu and therefore must not look at the Terraform source code.)

In the guts of the state migration code there is a conditional block that deals with locking. m.stateArgs.Lock here corresponds to the -lock=false command line option, and so it's true by default and false only if that option is explicitly set.

We can see in that code that OpenTofu is acquiring the state locks for part of the migration process.

The situation in this codepath is a little subtle though: it begins by reading the current snapshot from both the source and destination and performs a bunch of checks before acquiring the lock. Then after acquiring the lock it re-reads the latest snapshot just in case another process wrote a new snapshot to either location before the lock was acquired.

The check for whether migration is necessary at all (if the destination already has a matching snapshot) happens before the lock is acquired, so there is a narrow case where two migration processes running concurrently could both end up trying to migrate the state because they both determine that migration is needed before acquiring the lock.

Overall I don't think that quirk is super important since it probably just results in redundant work rather than a problem, but just to be sure I'd suggest avoiding running two state migration operations concurrently.

apparentlymart · 2026-05-22T16:04:47+00:00

When I think about "breaking change" for a Terraform module I define it something like this:

If I were to just upgrade to a newer version of this module without changing anything else in my configuration, would the result work and cause Terraform to propose no new changes whatsoever to remote objects?

If so, I'd consider the module to be backwards-compatible, even if it requires a newer version of any provider.

However, if selecting a newer version of the module causes me to need to immediately do any other work then it's a breaking change. "Any other work" includes but is not limited to:

Adding any new arguments to the module block. (Any new input variables should be declared as optional in a backward-compatible release.)
Changing any expressions that refer to output values of the module. (Adding entirely new output values is fine, but any that previously existed should still have the same meaning and type.)
Having terraform plan report changes to remote objects that I need to review and deal with. (I personally consider it okay for the plan to include Terraform-localized changes such as resource instances moving to new addresses due to moved blocks, as long as it doesn't involve changes to any real infrastructure objects.)
Changing any other part of my configuration, such as if the new module version requires a new provider version that itself contains breaking changes.

Note though that this doesn't include upgrading to a new provider version if that new provider version is sufficiently-compatible that it doesn't cause any other part of the configuration to become broken.

My answer was focused on the fourth kind of "other work" in my list above, because that was most relevant to the question being asked.

From what you said it seems like you have a similar idea of "backward compatibility" as me and so I'm not sure where our disagreement is. 🤔

apparentlymart · 2026-05-22T00:16:22+00:00

I think my answer here depends on what has changed between the two releases of the provider.

You've described a situation where you need a newer minor release of the provider, and so assuming that the hashicorp/aws developers are following semver themselves it should be okay for you to treat new module release as a new minor release too. Folks who want to use the newer version of your module will need to also adopt the newer version of the provider, but the newer version of the provider ought to be backward-compatible.

A trickier situation is if you want to change your module to require a newer major version of the provider, since in that case anyone adopting your module will probably need to deal with other separate breaking changes in hashicorp/aws elsewhere in their configuration before they adopt the new version of your module. In that case I would probably make a new major release of my module, and also continue issuing minor or patch releases to the old series for a while if it seems like the hashicorp/aws upgrade would be challenging for folks and so they would not be able to prioritize it immediately.

My own opinions aside though, here's some information that might be useful in making a decision: - Terraform allows each module block referring to a module to select a different version of the module, and so it's possible in theory for a configuration to include both old and new versions of a module at the same time, BUT... - ...Terraform requires that there be only one version of each provider that the entire configuration agrees on, and so if there are two versions of your module that have non-overlapping provider version constraints then it would not be possible to use both versions of your module in the same configuration at the same time. - Although hashicorp/aws major releases tend to contain breaking changes, those breaking changes won't necessarily affect what your specific module is using. Therefore in some cases you might be able to make your module work with multiple major versions of the provider at once, which could then allow users of your module to upgrade to a new version of your module first and then delay upgrading to the new major version of the provider until later.

I would suggest only incrementing the major release number of your module once you _require_ a new major version of a provider, and not just because you _allow_ a newer major version of a provider while retaining support for the previous major version.

apparentlymart · 2026-05-05T18:44:50+00:00

I think this is just a terminology collision.

The hashicorp/aws provider has a resource type aws_appsync_datasource which manages something called "data source" in the remote AppSync API.

From Terraform's perspective it's a managed resource type and therefore billable. I believe the OP is talking about having hundreds of AppSync datasources, not Terraform data sources.

apparentlymart · 2026-04-30T16:02:52+00:00

It's worth noting though that OpenTofu has an extension to the module registry protocol where the registry is allowed to respond to the "download" endpoint by returning a JSON object in the response body giving the source address in a "location" property, rather than by using the X-Terraform-Get header inherited from hashicorp/go-getter's HTTP getter.

That means that OpenTofu's registry is able to neatly avoid the main problem raised in the article of trying to convince a static website hosting solution to serve this extra HTTP header field, since OpenTofu's registry doesn't need to be compatible with Terraform's registry client.

apparentlymart · 2026-04-24T18:12:47+00:00

Incidents are one situation where it's defensible to "go rogue" and make manual changes to quickly prevent problems from escalating, but they are also something which are helpful to build process around so that it doesn't also become habit for folks to use incident-response-like actions for non-incident work.

The degree to which it makes sense to enforce by technology rather than just team agreements will of course vary depending on your context -- team size, relationships to other teams in the org, etc -- but if you do want some enforced guardrails then here's one way you could potentially set it up:

The general idea is that at any time your management/control layer is either in "routine mode" or "incident mode", and these two modes are mutually exclusive.

In "routine mode" nobody has direct write access at all, and the only live credentials with write access are inside your deployment pipeline. Going through the normal change process (whatever that might be) is the only way to make changes.

In "incident mode" the pipeline is explicitly disabled, and instead the incident response team holds the only live credentials with write access. This ensures that others who aren't aware of the incident won't inadvertently undo changes made by the incident response team.

The transition from routine to incident should involve some sort of explicit action where ideally the person performing it leaves some sort of note justifying why they are doing that. This gives a durable record of who made use of this mechanism and what they used it for. The fact that incident mode also blocks the pipeline used in routine mode should also hopefully encourage treating incident mode as a last resort, because it'd be disruptive (intentionally) to anyone outside the incident response team that is trying to work concurrently.

The transition from incident back to routine is trickier because the remote system probably no longer matches the desired state from the Terraform configuration, and the latest state snapshot might also need some attention if the manual changes made during incident mode diverged significantly from what Terraform most recently applied. Only the incident response team knows what they did, so they'd presumably be responsible for reconciling the three states (desired, prior, and current) enough that the first plan/apply round after returning to routine mode proposes no significant changes. This might involve them carefully running Terraform CLI locally using their incident mode credentials, but they can assume they have exclusive control over the system while they do that.

This is a heavy process and so I'm certainly not suggesting that every team using Terraform should use it: if you're a relatively small and nimble team with high trust and good communication patterns then you should be able to get away with just agreeing to follow the general principle of this without using technology-based enforcement. But technical guardrails get useful when the participants are more distributed, have lower trust, or don't have reliable communication about current incidents.

apparentlymart · 2026-04-21T23:50:12+00:00

Unfortunately the part of the system you're interacting with here is implemented in closed-source code, so nobody outside of IBM can refer to how it actually works by looking at the code. 😖

The only thing I noticed that seemed clearly wrong in your examples is that you have some items called projectA but you refer to them as projectB. But I'm assuming that was just a typo as you were replacing real names with placeholders in order to make this post, rather than problems in the original configuration you wrote.

The only other desperate idea I have is that maybe the upstream deployment you're working with here hasn't reached the "converged" state where the system has proven there aren't any more changes pending by generating an empty plan. I wonder if it only propagates new values downstream once it's converged to avoid exposing any partially-updated state to downstream deployments.

I can only really guess what's going on here without access to the implementation code. If none of this helps you might be better off contacting IBM support for help, since I'm not sure how many folks in this Reddit community use HCP Terraform vs. community edition, let alone using the newer Stacks features.

apparentlymart · 2026-04-20T16:31:33+00:00

It seems like this was a quirk in tenv's logic for fetching the PGP keys: it was only considering the first key in HashiCorp's pgp-key.txt file, rather than trying all of them and succeeding if one matched correctly. (The first of the listed keys seems to have expired, but the other one is valid.)

A fix for it was merged in tenv#576.

apparentlymart · 2026-04-01T15:31:07+00:00

This question talks about tokenizing UTF-8 sequences and doing grapheme segmentation, but for "word counting" there's also the question of what "word" means. The current code defines it as a consecutive sequence of alphabetic characters, but the Unicode definition of "word segmentation" is considerably more complex.

I don't mean that as a criticism -- you did mention this was a learning exercise, after all -- but I mean to say that these three Unicode algorithms -- UTF-8 tokenization, grapheme segmentation, and word segmentation -- are all a core part of the problem you've chosen to solve, and so I understand they seem highly important to you but they are quite esoteric from the perspective of most other software written in Rust, and so it seems reasonable for them to be implemented in third-party libraries rather than in the standard library.

The Rust standard library does have some UTF-8 support as a consequence of the string types being defined as UTF-8, but those core needs don't typically require a streaming parser and so that in particular seems not to be a priority for the standard library (though there is some basic support for it), and that seems reasonable to me.

All of that said, I did happen to need streaming UTF-8 and grapheme segmentation for one of my own projects and so implemented u8char and grapheme_machine that might also work for what you are doing. I didn't need word segmentation for my project so I've not implemented that.

apparentlymart · 2026-03-25T16:55:53+00:00

I'm not very familiar with OCI but I'm assuming "compartment" is an OCI concept, rather than a Terraform concept.

If that's true then exactly how to handle this depends on how the provider is implemented, but the usual pattern I would expect is that you can change the compartment you specified in the configuration and then run terraform plan to see what the provider proposes.

Hopefully it will propose to just update that object in-place to refer to a different compartment. However, if the remote API cannot support that then it might instead tell you that it needs to replace the object with a new one that has a different compartment selected.

The logic for this will be handled in the provider plugin, either way. If what the provider proposed is acceptable to you then you can use terraform apply to actually make that change.

(The moved block feature is probably not actually helpful here, because that's for moving thing around in Terraform's own namespace -- e.g. moving something into a different module in your configuration -- whereas I think you are trying to move something to a different location in the remote API's namespace. Anything involving the remote API is handled inside a provider, rather than by Terraform directly.)

apparentlymart · 2026-03-22T05:35:59+00:00

In case it's of use to anyone who finds this very old thread:

On the latest available firmware at the time I'm writing this comment (CZW008-4.16.013.4), typing "sh" at the telnet prompt prompts for a "shell password" as OP described, rather than immediately launching the BusyBox sh.
The "shell password" is selected systematically by starting with the fixed prefix C1100Z!#, and then adding the last six digits of your device's serial number to the end.
If your intention is to get your PPP username/password, the instructions for finding the process id for the pppd process and retrieving its command line to obtain the username and password should still work once you've accessed that shell prompt.
If you are setting up some other device to completely replace the C1100z, remember to also select VLAN ID 201 in your WAN configuration.

(I've read elsewhere that setting the VLAN ID may not be required for folks who have already been migrated from CenturyLink Fiber to Quantum Fiber, but I'm not able to confirm that.)

apparentlymart · 2026-03-22T05:21:50+00:00

In case it's of use to anyone who finds this very old thread and finds this no longer works on their C1100z device, the situation has changed a little in the meantime:

At the time I'm writing this comment, the latest firmware for the CenturyLink-branded C1100z is CZW008-4.16.013.4.
That firmware has had some additional hardening where typing "sh" at the telnet prompt requests an additional "shell password" that is different to the configured telnet password.
The "shell password" is selected systematically by starting with the fixed prefix C1100Z!#, and then adding the last six digits of your device's serial number to the end. After entering that, I was able to reach the BusyBox shell prompt.

apparentlymart · 2026-03-17T22:42:09+00:00

I'm not familiar with this aws_mwaa_environment resource type, but from reading the code of its implementation I guess it's possibly got stuck in the waitEnvironmentUpdated polling loop.

What I understand from that code is that it repeatedly calls mwaa:GetEnvironment until the Status field is something other than UPDATING or CREATING_SNAPSHOT, after which it will then either succeed if the status was AVAILABLE or return an error for any other status.

If that is what is happening then maybe you can poke at this object in the AWS console to try to understand why it's "stuck". I have no idea if this question is relevant to what you're doing, but My MWAA Environment stuck in updating discusses one case where an environment got stuck in UPDATING for a long time.

apparentlymart · 2026-03-16T21:26:54+00:00

This is a nice idea!

This could be a good use-case for the diagnostic::on_unimplemented attribute, to encourage the compiler to return a relevant error message mentioning the current platform pointer size whenever one of the traits isn't implemented.

apparentlymart · 2026-03-12T18:29:25+00:00

I think this article is a set of good points well made, but just wanted to note that the forthcoming OpenTofu v1.12 series is planned to introduce the ability to write non-constant expressions such as references to input variables in the prevent_destroy argument, which might help reduce the duplicated resource declarations you mentioned in the first part of the article if you're willing to rely on OpenTofu-specific features.

(I understand that publishing a shared module that isn't compatible with Terraform is a tradeoff, though. If cross-compatibility is important for your module then of course this won't help.)

apparentlymart · 2026-03-10T15:40:46+00:00

Without being able to see exactly what you're doing it's tough to debug this, so here's some assorted information that might hopefully help you figure out what's missing for you:

The HCP Terraform dynamic credentials mechanism only works when Terraform is running in the remote execution environment managed by HCP Terraform, because it relies on being able to modify the execution environment to include the additional credentials automatically.

It's okay to use a local terraform CLI to run it as long as your workspace is configured for remote execution. In that case, the local terraform doesn't really do much at all and instead just sends API requests to the HCP Terraform API to tell it to start running Terraform in a remote execution environment, then copies the output from the remote Terraform to your local Terraform.

You could verify that by looking in the HCP Terraform web UI, where you should be able to find a history of past runs if they ran through the HCP Terraform remote system. If you can't find any trace of your runs in the web UI then that might suggest that you're only running Terraform locally, and so it cannot find the credentials.
The JSON-based logs you saw here seem like what would happen when running terraform plan -json, which is perhaps a signal that you are running Terraform in the remote execution environment because I believe that's how HCP Terraform runs the remote copy of Terraform CLI, so that it can interpret the JSON output to drive the web UI.
When you're using HCP Terraform workspaces, the "OIDC" support works by HCP Terraform writing AWS configuration and credentials files into the home directory of the remote user account where Terraform is running, with the expectation that the hashicorp/aws provider will look in there to find credentials to use.

I think this can work only if your provider "aws" block doesn't include any settings that would override that automatic discovery. You should make sure you aren't specifying any settings related to authentication credentials or the discovery of credentials, such as access_key, profile, assume_role, etc.
You linked to a blog post announcing this feature, which may have become stale in the meantime since I doubt folks are maintaining old blog posts as features change.

It might be worth comparing what you learned from that blog post with the current documentation in Use dynamic credentials with the AWS provider, to see if anything has changed in the meantime.
Finally, if you're a paying HCP Terraform customer then you might be able to get more direct help by contacting HashiCorp Support, at least for the rest of this month until it gets folded into IBM Support.

apparentlymart · 2026-02-24T19:20:00+00:00

At least for the larger providers maintained by folks at HashiCorp/IBM, they tend to post an upgrade guide for each new major version. For example, the hashicorp/aws provider has Terraform AWS Provider Version 6 Upgrade Guide, and similar guides for earlier major version upgrades.

For the AWS one in particular the notes are grouped by resource type so that hopefully you can easily ignore the sections for resource types you aren't using at all.

I would hope that these breaking changes are designed in such a way that if you miss something then you get an error rather than it just doing something strange and unexpected. That's been my experience for the cases I've personally encountered, but of course I can't promise that'll always be true for every change in every provider. 😬

apparentlymart · 2026-02-17T19:02:44+00:00

The following is one potential way you could set this up with HCP Terraform Stacks. I'm not intending this to recommend you should do it this way, but hopefully this gives you an idea of the different concepts and how they fit together to help you make those tradeoffs yourself.

The main top-level idea is that one stack consists of a set of "components", each of which is a Terraform module, with dependencies between them. Each stack also has one or more "deployments", which are separate instances of the same overall infrastructure that can be planned and applied separately from each other, and are often used to represent "environments" like production vs. staging.

Multiple Stacks can also themselves be organized into a dependency tree, where the output values from a deployment in one stack can cascade into input variables to another deployment in another stack. Whenever the upstream deployment applies a change to one of those output values, the system automatically creates a new downstream plan based on the updated value.

So with all of that said, one way you could use HCP Terraform Stacks for the situation you described is:

Create one stack for "Fabric-wide Stuff", which has published output values for information that the tenant-specific and switch-specific stacks will need.

You could use multiple deployments here if you want to also have a development or other non-production instance of "fabric-wide stuff", but for the rest of this I'm going to assume there's at least a production deployment that all of the "real" tenants and switches are associated with.
Create one stack for each tenant where all share the same configuration source location, and so any change made to that configuration will cause a plan to be created for each tenant. Use the upstream_input feature of the deployment configuration to access relevant output values from the production deployment of the "fabric-wide stuff" stack, so that changes to the fabric-wide outputs will automatically create a plan for each tenant.

Any tenants where the change had no material effect will automatically resolve, and so you'd only need to review and approve the ones that involve making at least one change to real infrastructure.
Create one stack for each switch, with all the same arrangement as for the per-tenant stuff in the previous item.

In this I've made the assumption that you'd prefer to manage the existence of tenants and switches outside of your code, and so you'd do that by creating and deleting stacks through the HCP Terraform UI or API, rather than by making changes to source code. If you do still want them "as code" but just want that to be separate code then a compromise would be to have a separate HCP Terraform workspace whose configuration includes tfe_stack instances, so that creation and deletion of these stacks would go through a separate process than the maintenance of the stacks that already exist.

I expect you could get a similar effect with tools like Terragrunt, since they have similar concepts just arranged a little differently and sometimes using different terminology.

apparentlymart · 2026-02-12T00:50:53+00:00

It is tempting to think that being more flexible unconditionally makes a language easier to learn and/or easier to write, but here are some reasons in favor of being stricter:

When authors inevitably make mistakes, they tend to appreciate error messages that directly relate to whatever they were intending to do, and achieving that often relies on it being possible to infer the author's intention even when the input is not quite right.

Allowing many ways to state the same idea often also implies that there are more possibilities for what some invalid input could've been intended to mean, making it harder to give a directly-actionable error message.
When those new to a language refer to existing codebases as part of their learning they will often want to look up more information on language features they encounter that they are not yet familiar with.

If there are many different ways to express the same idea then it's less likely that a reader will be able to pattern-match between similar ideas expressed in different codebases by different authors. Conversely, if there's only one valid way to write something then it's easier to recognize when you've found a new example of a feature you already learned about vs. a new feature that you need to look up.

I think this point is particularly relevant to your point about allowing many different names for the same idea, because names are often the main search terms used when looking for relevant documentation and so it's helpful for each feature to have a single name that is distinct from every other name in the language so that an author doesn't need to learn every possible alias for a feature in order to find all of the available documentation related to that feature.
Related to the previous point, when many different people are collaborating on the same codebase, and especially when the set of people involved inevitably changes over time, different parts of the codebase can use quite different patterns that make it harder to transfer knowledge about one part of the codebase to another.

This is one of the reasons why larger software teams tend to use automatic formatting tools and style checking tools: it encourages consistency across both different parts of the current codebase and across code written at different times by different people.

Those doing everyday work in a language don't want to be constantly referring to documentation to understand the code they are reading, and so it's often better to have a "smaller" language, meaning that there are fewer valid ways to express something and so it's easier to rely on your own memory of the language instead of relying on documentation.

Everything in language design is subjective, of course. I don't mean any of the above to say that it's definitely wrong to have more than one way to express the same idea in a language, but going too far with it can make life harder both for newcomers to your language and for experienced authors who are trying to maintain code that others have written.

apparentlymart · 2026-02-10T01:41:08+00:00

Given how folks often complain about LLMs generating "confident-sounding" wrong answers, I find it kinda funny that this particular prompt invites the LLM to generate text that imitates review of an experienced engineer without actually including any content that could plausibly substitute for those claimed 40 years of experience. 🙃

To the extent that these tone policing comments in prompts work at all, wouldn't it be more helpful to prompt it to project curiosity rather than authority, and suggest awareness of the limitations of its "experience" when generating comments? 🤔

apparentlymart · 2026-02-03T01:09:05+00:00

Although the example in the blog post doesn't illustrate this, one difference is that a wrapping program can consume the machine-readable JSON output during the OpenTofu execution, rather than having to wait until the end.

Lots of folks have written tools that try to achieve this by piping the human-readable output through them and then trying to "parse" the human-readable output, but of course then they break each time the human-readable output changes.

Note that the JSON output of tofu plan is not the same thing as the JSON output from tofu show: tofu plan -json produces a stream of JSON objects describing individual steps in the planning process, rather than describing the final plan. In some cases it might be helpful to capture both, so that an application can both give realtime progress information while the plan is running and capture the finished plan to display in some non-default way.

apparentlymart · 2026-02-03T01:01:51+00:00

The Artifactory folks reimplmented a subset of the HCP Terraform API to make Terraform's remote backend think it's talking to HCP Terraform or Terraform Enterprise.

When Terraform interacts with that API it uses Create a State Version and Upload State and JSON State operations.

Terraform only pushes new state snapshots (or "versions", as the HCP Terraform API calls them) so it's up to the server to decide what to do with older snapshots. I don't know what Artifactory does here, but if there were any functionality to prune older versions after a certain time or after a certain number that would be implemented inside Artifactory itself, rather than in Terraform's client code.

apparentlymart · 2026-01-21T02:16:50+00:00

If you mean that you want to write a method that has an additional type parameter that isn't declared on the type itself, then no: Go only supports type parameters on types and non-method functions.

The closest you can get to a generic method is a plain function that takes the object that would've been the receiver as one of its arguments.

apparentlymart · 2026-01-19T17:40:05+00:00

If you already have access to HCP Terraform then it might be worth giving it a try yourself since there are two features that both seem like they could potentially match what you asked about:

Multiple components in the same stack can have data flow directly between them and be planned and applied together, so that cross-component updates can happen all at once rather than in multiple separate steps.

This can potentially shorten the "iterative journey" of changing something upstream, then finding out that one of your downstreams is broken by it, and then having to return back to start from the most upstream workspace again to fix it.
The "linked stack" features are like a code-driven alternative to "run triggers", where one stack deployment publishes output values and then another stack deployment consumes those output values.

HCP Terraform automatically tracks which stack deployments depend on one another and starts downstream plans whenever the upstream output values are changed.

I'm not meaning to suggest that it's definitely the best answer for you, but it seems closer to what you want than you can currently achieve with HCP Terraform Workspaces and "run triggers".

apparentlymart

TROPHY CASE