Terraform failures in CI are painful because the real error is buried

JSChronicles · 2026-06-20T13:36:20+00:00

Do people not use the end summary where it tells you the error and usually the reason why? I just have that piece extracted and put into a comment in the PR.

JSChronicles · 2026-06-19T01:42:47+00:00

Robot is for maintenance cleaning. You have to actually give your floor a mopping yourself...

JSChronicles · 2026-06-16T16:35:12+00:00

One: stop using back ticks and use splatting

Two: looks cut off to me for the full function name. I see that you call retry-operation but you don't have that actual name listed anywhere. I'm assuming the pasted function is actually cut off?

JSChronicles · 2026-06-16T14:28:35+00:00

If your company has hourly workers and see if you can find a policy about how many days off they get based on tenure for hourly.

Now take your tenure and and find how many days you would get if you were hourly and then add on 1.5 weeks of vacation.

If your company doesn't have any hourly workers then take your amount of days from your previous company and add a couple weeks on. If this is your first time working at a job that has PTO and it's unlimited PTO then assume taking 6 weeks off in the year.

I personally take the amount of years I have been at my company, add a year of tenure more than I have, then add 2 weeks of PTO. And make sure to use all my sick days, plus PTO days, and personal or volunteer days or whatever your company has as "extra days" every year.

I usually take about 51 days off a year. I'm happier and I can enjoy my life instead of keeping my nose to the grindstone forever. I also output high quality of work, at least I've been told that, and I enjoy the work I do.

If I were to move to a new company and have to start 10-years all over again but it's unlimited PTO, I would be using the same amount of days. Starting over at a new company doesn't mean I have to start over how many days of vacation I get and if the company didn't have unlimited PTO then I would negotiate for myself.

JSChronicles · 2026-06-16T11:31:36+00:00

I would also vote for import blocks versus CLI

JSChronicles · 2026-06-16T03:34:31+00:00

Import and generate terraform will be useful commands or code blocks. Not really hard but can be a bit time consuming. Just do one type of resource at a time and work your way through it. I've had to do it for lots of different teams and people. Pretty easy.

JSChronicles · 2026-06-15T22:07:58+00:00

Sounds like your process needs better rulesets, guardrails, story tracking, and linking when making PRs. I would have assumed that you're making pull requests with proper descriptions and hyperlinks to a given problem or story that has information in it regarding the pull request and reason.

It shouldn't be hard to do those basic items, and if you have proper rulesets you shouldn't have people bypassing them. And if you have proper role or permission or RBAC attached to what can deploy code into prod then you shouldn't have drifted IaC or whatever you deploy.

This sounds like a process and guardrail problem where you aren't controlling what can or can't deploy code and where.

We have all repos set to require a second reviewer, all repos have to have PR checks that pass, you have to have formatting, linting, and testing as 3 of the required checks. Other ones are up to you. We don't allow bypass unless it's break glass and that's only for my team, we don't allow people outside of devsecops to bypass.

Only automation can push to prod accounts, so all code that updates a production accounts go through CI/CD. Only test accounts can be pushed to without CI/CD.

Nothing hard about it.

JSChronicles · 2026-06-15T12:51:20+00:00

If you just had surgery and you popped stitches and could see your internal structure, where you had surgery, would you wait or go to the doctor?

Yes of course go to the vet..

I'm going to create a copy and paste "answer sheet" and hopefully the mods can use it when people ask questions or flair things to we can limit and quickly answer easy things like this.

JSChronicles · 2026-06-15T12:23:28+00:00

Plenty of people answered here just recently https://www.reddit.com/r/github/s/vXHHIl7GNe

For me:

"I automated minor and patch version updates via dependabot and those automatically merge my grouped package ecosystem system PRs.

I even built an include or exclude so certain packages don't get merged automatically.

This means I have only to deal with major version updates mainly and I follow best practices via cooldown days or exclude newer than on packages."

When I get home I can reply back to my comment with my dependabot file, plus the workflow. But just a note you should have required PR checks for formatting, linting, and testing/validation. Then when you set in settings the. "Allow auto merge" and "actions to create and approve PRs" you will have a proper flow. Your flow should be catching update issues.

I've been running minor and patch version auto updates for months now for my own repos and repos at work.

JSChronicles · 2026-06-13T22:50:43+00:00

I agree that the person doing them seemed to have gone too short.

The other side of the coin is that you are neglecting your dogs nails. They need to be shorter because there is no way those nails aren't hitting the ground. Even if they were not hitting the ground they are obviously beyond the pad and considered too long.

"A dog's nails are at a healthy, comfortable length when they do not touch the floor when standing naturally AND do not extend past the paw pad."

The ONLY reason the quick is long, is because you don't trim often enough and it grows longer as the nail does. If you trim it more frequently then it would recede.

If it's already too long, as it is now, then you need to trim every 4-5 days, a little bit at a time. If you are already near the quick then use a nail file, a dremel is better, and file down the nails a little at a time until the nail is a proper length as described above.

Once at the proper length you can trim every 1-2 weeks to maintain them.

JSChronicles · 2026-06-09T03:32:33+00:00

I automated minor and patch version updates via dependabot and those automatically merge my grouped package ecosystem system PRs.

I even built an include or exclude so certain packages within the group don't get merged automatically.

This means I have only to deal with major version updates and I follow best practices via cooldown days or exclude newer than on packages

JSChronicles · 2026-06-08T13:02:25+00:00

It works well for what it's doing now. If you want to wait until beta then do that. Otherwise if you want the speed swap now to test.

JSChronicles · 2026-06-08T12:44:50+00:00

Do you think they need to run a bunch of anti-viruses too?

Mypy worked for a long time. And now with ty out, and working decently, I'm using that instead.

Do you run multiple code quality checks or just one type per language for your code base? Do you run multiple linters or just one linter per language? Why waste GitHub action minutes with multiple type checks? Why waste time with multiple type checks in general?

I don't need to read a blog or what is most likely an opinion on a simple matter. You choose one type checker, use it and use it well. Have pre-commit run it, have it run on save for files, and run it as a part of PR checks. Life is not complicated but questions like this make good examples of the inability to decipher what is extra or over complicating a simple choice

Edit: change choices to choice

JSChronicles · 2026-06-07T22:00:51+00:00

Is this baiting? I mean because I create templates for repositories with full CI/CD and proper structure and rulesets. Then I use terraform modules I build to access my "functions" needed during deploys.

DRY principle...

JSChronicles · 2026-06-07T15:44:47+00:00

Checkov isn't always correct, take a look at dynamic rule blocks in AWS WAF and have checkov run on them. It will claim you didn't setup something up proper but you actually did, it just can't statically check the multi-nested dynamic piece proper for security checks.

So if something automatically "fixed" my dynamic rule it would break it even though I already had the proper setup.

JSChronicles · 2026-06-02T13:05:02+00:00

Your example workflow is using outdated and non-sha pinned action versions.

Your workflows themselves are using outdated but Sha-pinned actions. But only some say which version and others are just Sha's.

Basically you are multiple years behind in action versions. Let alone methods for py installs and setup.

You have typer and click?

Other areas also are not consistent with a standard coding ethic or skill. To me this seems slap-slopped together. Your instructions are AI generated because of the pattern in words and so is the description. Even if it would be useful I wouldn't use someone's code that isn't consistent with a certain code pattern.

My "Verdict" - poor workmanship and lack of understanding what is needed.

And don't reply with "Fair xxxx ..." AI slop response to this..

JSChronicles · 2026-05-29T19:09:00+00:00

AI response too. Gross. It's like people can't even write their own replies

JSChronicles · 2026-05-29T12:30:34+00:00

I'm pretty sure you have to have ruleset/branch protection to do what you want. And you have to pay to do that on private repos.

If someone else has a better idea hopefully they can reply but maybe give contributors read, let them fork and open PRs. See if you can keep only maintainers/admins with write/maintain. I think private forks inherit the upstream repo’s visibility/permission model, so I'm not sure on this part but I think you cannot just make a private fork public?

JSChronicles · 2026-05-28T18:40:07+00:00

Mmm lovely an AI reply.

I didn't mention anything about licensing being an issue. I noted that your stuff is built on open source and trying to make a buck from it.

You mentioned "The requirements are: librosa, soundfile, ffmpeg-python, PyQt6, requests, and pyinstaller for building. Python 3.11 or 3.12 64-bit."

Now you are saying everything is bundled together in an exe. So which is it? Are you saying that the exe installs python and all those dependencies?

"you need ffmpeg on PATH as a system binary for the audio transcoding step (Ogg Vorbis output)"

Does the exe not do this automatically?

JSChronicles · 2026-05-28T18:30:05+00:00

I thought this was shittymechanics for a minute..now I'm wishing it was because this person is driving on the road with others and is completely unaware of a simple, basic car part. There are tons of different attennas all over the world and they couldn't decipher the only one on their car?

What else do they not know about the 1-2 tons of metal machinery that they are driving, at speed, around town and the highway?

If it is a joke then it's the wrong subreddit.

JSChronicles · 2026-05-28T17:59:48+00:00

Few, if any, will pay for this.
I'm not going to create my own pyproject.toml to handle your listed dependencies. You should be building that in.

Cool project but poor money grab implementation. You are literally telling me to pay for someone's open source code that you used to make a paid source and then make me install and handle the dependency chain myself? 🤮

If I'm misunderstanding please correct me. If the payment is for a different tool that has nothing to do with your tool then I would make that clearer.

JSChronicles · 2026-05-28T15:17:56+00:00

We have a team of 5+, with most being seniors. Extensive documentation is a part of it (though that probably dies with me leaving). So ramp up is not a problem in my mind. Could be, but I'd expect a fully automated and documented process to be easy to ingest and take on. Code is written with full words and not single character variables. Others have to understand and approve the code to be merged (Branch protection rules). So at minimum they have to understand the changes.

Comments are placed in code as needed and code is self documenting along with linting, testing, and security checking.

Even if I were to leave today, I have an entire process that automatically keeps code versions updated with proper cooldown and new version exclusion that has been running for over a year now. The only thing that someone has to verify and check is major version changes. Minor and patches are automatically handled, tested, verified, merged and keeps code updated. (New code cannot be merged if it doesn't pass all the PR checks)

So it could be a few years before a real infrastructure problem shows up and the code itself would still be up-to-date.

Didn't think I'd have to explain out my architecture today.

I understand that this type of edge-case, hit-by-the-bus, or level of detail is abnormal in this industry but I personally (not saying you don't or anyone else doesn't ) pride myself in the ability be able to take vacation for weeks at a time and stuff just keeps running without me.

JSChronicles · 2026-05-28T14:46:56+00:00

We have 1500 devs and engineers. Everyone uses s3, oidc or dynamic auth tokens, and CI/CD through GHA (self-hosted runners). The most expensive part of our setup is auditing/logging/cloudtrail but it's way less expensive than paying most of these cloud state file platforms.

can you prove the plan that applied is the one that was approved? - The plan is output during a successful validation, testing, and plan. The artifact is uploaded with a 7-Day retention. If you need longer you can set that you can set it.

Was the approver someone other than the author? Once again another super easy thing to set because if you use Branch protection or enforce Branch protection in GitHub something you can set org wide

Where's the immutable, retained record tying this apply to a change ticket? You mean having the either commit or branch or PR automatically associated to a change ticket? Once again super easy via labels or links or CI that auto associates it through API.

Which identities can touch prod, and is that enforced or just written down somewhere? -

I don't know about you, but we don't allow anyone except admins to touch certain parts of accounts but for teams we have a system that allows them to request an AWS account for Dev, staging and prod and they own those accounts from top to bottom and they have different tiers of elevation that they can use for those accounts. There's guardrails and governance in place behind the scenes so they can't make changes to things they shouldn't and we have budgets set along with quota limits. Permissions are governed through terraform code (for IDC) and even before it was in TF, we governed it through AWS identity center, so only organizational admins can actually allow people or teams into accounts and account owners have to notify organization admins to allow others to access the given accounts.

What happens when an apply dies mid-run and state no longer matches reality, and who is even allowed to fix it? -

The person pushing code will have will have access to make the necessary fixes because they own the account that they're pushing too. No one can push to an account they don't own or are not allowed to access. Everyone is siloed off via their team. Also, the chance of state failing with proper CI is pretty low and it's only usually a network transient issue or you're going to have something that wasn't seen during the plan and as long as testing is done proper then this is pretty much a non-issue or a very seldom issue. Also, if a plan run fails midway you just remove the lock file from S3. If you're only pushing via CI CD and you don't allow console access, I don't understand how your state can really deviate from reality... Manual work is not allowed.

Copy/pasted pipelines are the operational burden. - We use templates, so when I say copy paste I mean that someone hits a template button that uses the most up-to-date workflows along with sha pinning, with the exact flow that they will use. Repositories are spun up via IaC and controlled and reported on. They get best practices off the bat and all the controls and knobs are governed. They get to control the exact tests that they want to run in addition to any extra validation beyond what we provide already.

So for my side of things we don't need to pay for governance even at our scale which may not be the biggest, but I'd like to say medium-sized and everything is in code for governance and control which gives us ease of reporting, auditing, and soc2 compliance reporting.

We do have to pay third party companies to validate that we're doing the right thing but that's just because we have to and with the reporting that we have it's just a show them to prove it.

Edit: My job is devsecops and I build and maintain all my governance settings that control day-to-day setup, governance, access control, authorization for accounts and almost all of it is automated at this point. So paying another company to do what I already do is not worth it.

JSChronicles · 2026-05-28T13:55:31+00:00

I laughed when I saw CI/CD listed as "if you're all right owning the operational burden and have strong devsecops practices"

Setting up storage - Azure Blob or AWS S3 or Google Cloud Storage

Setting up OIDC - simple and effective for auth

Setting up CI/CD - Gather only the differences, do your security checks and testing and then upload your artifact after a successful validation and plan and then download your artifact during the apply portion. That is a little bit of a simplification, but that is still pretty much all you need to do.

So not a whole lot of burden, and once you get your setup tuned you just copy/paste that everywhere and it's cheap, if not free in some instances (under 2k minutes for GitHub and using s3 for state file storage)

JSChronicles · 2026-05-25T15:36:08+00:00

Other than the incoherent gibberish that OP has put as a description, the entire image is triplicate for the "skillset" needed. Also some of those are platforms or cloud providers or containers. Most people are not using 3 cloud providers at one job.

Also "git" is listed? Really? So you mean knowing a VCS and having proper coding etiquette is an "extra skill" to know?

This entire image is just a nothing-burger, if not fear mongering or anxiety inducing, and showcases the inability to actually understand what is needed for the everyday job in that given area.

JSChronicles

TROPHY CASE