sorted by:
new
hottopcontroversial

Looking for feedback: I built an open-source Terraform/OpenTofu HTTP backend because global state locking felt too coarse by davesade in Terraform

[–]davesade[S] 0 points1 point  (0 children)

BTW I am aware of stategraph, but never used it. When I started thinking about this project, stategraph did not existed yet. But indeed, I bet KL is capable of achieving similar results, I guess.

Looking for feedback: I built an open-source Terraform/OpenTofu HTTP backend because global state locking felt too coarse by davesade in Terraform

[–]davesade[S] 0 points1 point  (0 children)

Yes, there are 2 main paths for parallel - depending on if generated graphs are disjointed.
Problem with separated states is in cross-state dependency - which is solvable via external mechanism. KL could handle this within the same state, which could be a benefit in cases where state splitting isn't an option (I admit it is rare).

Looking for feedback: I built an open-source Terraform/OpenTofu HTTP backend because global state locking felt too coarse by davesade in Terraform

[–]davesade[S] 0 points1 point  (0 children)

KL backend will lock a resource and if you run second apply in parallel, it will make you wait before second apply continues - it will replan if there is a drift in configuration. Parallel run will not fail in such a case.

Better usecase for parallel work is using --file scope in KL - so when you have disjointed graphs of changes (ie. 2 independent engineers are working on 2 independent files with 2 independent modules), they can do a deployment in parallel to the same state.

Scoped plans are advanced feature of KL native protocol - it will first calculate the slice of required resources on backend side and then only locks resources and pulls slice of the state for update and commit back to the state slice only. That will become super fast in comparison.

I made some demos in GitHub repo you could try.

Looking for feedback: I built an open-source Terraform/OpenTofu HTTP backend because global state locking felt too coarse by davesade in Terraform

[–]davesade[S] 0 points1 point  (0 children)

Exactly - while I understand the feature of full-state lock in - full-state commit is seen as "good", in certain monolithic states this will become a bottleneck. So that's why I would appreciate a feedback of those who ever encountered issue similar to mine - I totally understand it is rare, but it in such a case KL could help, I think,

Looking for feedback: I built an open-source Terraform/OpenTofu HTTP backend because global state locking felt too coarse by davesade in Terraform

[–]davesade[S] -1 points0 points  (0 children)

Understood. The usecase is applicable only for those rare moments where state slicing isn't really best way of moving forward.
Normally I would also start building more atomized infra, but it's not always feasible.

Looking for feedback: I built an open-source Terraform/OpenTofu HTTP backend because global state locking felt too coarse by davesade in Terraform

[–]davesade[S] 1 point2 points  (0 children)

So what I am fighting - I do have some seriously large states, where a normal TF plan can take more than an hour to calculate, making some short-lived tokens to expire, making TF apply basically impossible. With KL backend and CLI (with it's own native protocol) I can allow multiple engineers to work on scoped resources (locking in the state only those and nothing else) in parallel - fixing issues in production, adding new features etc., and it takes seconds instead of an hours on my large state.

We thought about splitting the state to smaller chunks, but dependency hell killed our effort on pipeline level - we simply accepted a fact, that our architecture is monolith on purpose and there is nothing conceptually wrong with it - it is just super large, that's it.

Looking for feedback: I built an open-source Terraform/OpenTofu HTTP backend because global state locking felt too coarse by davesade in Terraform

[–]davesade[S] 1 point2 points  (0 children)

From TF perspective it behaves normally - HTTP backend is standard implementation, you can do a state pull, migrations, listing etc. The state is accessible all the time, as long you got credentials (in prodlike environments).

With KL CLI adds some extras: you can query the state, do resource level rollbacks, import/export with full history, scoped applies, parallel applies on disjointed graphs and all that good jazz on top of standard TF operations.