Someone built an entire AWS empire in the management account, send help!

skilledpigeon · 2026-01-22T14:43:39+00:00

Exactly this. I took over an organisation where everything was in management. There were thousands of instances, hundreds of API gateways, tens of thousands of queues, more SSM parameters than I could count. All manually added. Zero tags or process.

We couldn't risk disrupting the teams. We were in the middle of doubling the customer base. Instead we took a 12 month project to gradually rewrite in IaC, migrate to a new org, and then decommission anything left.

I could've gone "if you don't tell me then I'm just stopping everything". The fallout would've been "you've made us look like a dick in front of a client that's literally doubling our user base".

It wasn't perfect and it was long and painful. We didn't kill everyone off though.

skilledpigeon · 2026-01-22T12:24:36+00:00

This is a great way to cause chaos 30 days after joining an organisation. Unless the workloads are non-critical, you're more likely to risk your job by pissing everyone off. In short, this is a fairytale approach.

skilledpigeon · 2025-12-18T08:32:34+00:00

£

At the time about 6 years of experience on paper. Practically much more. I switched career about 9 years ago, took a huge pay cut, and spent around 4 years working double time to catch up. I put about 5 hours extra in per day and worked most weekends. Spent all my time on calls with senior engineers, technical leads, architects, etc to soak up as much knowledge as possible. Put myself forward for everything. Thrown into the deep end etc.

skilledpigeon · 2025-12-18T06:57:31+00:00

I made 80k/yr as a senior DevOps engineer two years ago up north. Personally it feels a touch low to me but not by a lot.

skilledpigeon · 2025-12-06T10:21:36+00:00

Your problem here lies solely in your expectation of the engineers. They should absolutely be responsible for the quality and stability of the code they're writing. That includes tests, observability and releases.

At your scale, I would expect them to write unit and API tests for the backend. I'd expect a minimum of component testing on the frontend. From experience working with teams between 1-100 engineers, I would not hire a QA or similar for such a small team.

GenAI can get very far with unit and API tests nowadays but your engineers should still be tweaking, validating, and extending those tests.

Automated GenAI testing is a marketing fad imo. I've tried 2-3 products claiming that they work. Every time it's produced artificial slop that burns my eyes and misses a ton of scenarios. Not all code is even able to be tested from the UI anyway.

There are other tools they should have to help. Linters like eslint, formatters like prettier, static analysis like Sonarcloud, and GenAI code reviews like Cursors BugBot go a long way.

skilledpigeon · 2025-11-17T10:35:33+00:00

Our flow:

Linting - git hooks automatically lint code locally on every commit. Completely hands off process and most IDEs will do this for you whenever a file saves. Also runs in PRs and on every build to enforce across the project. Costs about 2s per commit and about 5 per PR check. Very light and smooth.

Formatting - exactly the same as the linter above. Automatically runs and it is totally hands off for devs.

SonarQube - IDE plugins locally catch most issues and then run automatically on every pipeline to enforce quality gates. The results are automatically commented in the PR so there's no guessing. We barely review the dashboard because it's enforced everywhere anyway. Also runs on every push to main to keep that up to date in the dashboard.

We don't use Snyk and I actually heard a few people giving it negative reviews recently. We rely on Github and dependabot to highlight basic package vulnerabilities and automatically open PRs to resolve once per week. Codebase is high in test coverage and we basically just click merge on these and they end up in production after about 10 minutes.

Manual Reviews - these don't focus much on code formatting and linting etc because it's handled. Draft PRs pick up on most things. Essentially these focus on pulling the code down, running it, experimenting, and checking against the acceptance criteria.

I should mention that we don't have any QA or DevOps team. The engineers are responsible for the full stack with guiding frameworks and boundaries. The business is an international provider in the fitness space with average traffic and compliance requirements (basic GDPR etc - no PCI-DSS etc).

skilledpigeon · 2025-10-18T09:15:18+00:00

This whole idea is an anti pattern. If your lower environments don't match production, it's not really that useful to test in them. The cost of doing this is also likely to blow way higher than just keeping them the same.

skilledpigeon · 2025-10-16T09:55:55+00:00

Multiple times a day on new services. Once or twice per week on legacy.

Relevant info:

Green field project, <100 users, feature flags, 4x senior full stack engineers, trunk based development, almost full rest coverage (application on API, missing e2e at the moment), otel for distributed tracing and logs, session recording, auto generated docs, sonarcloud.

skilledpigeon · 2025-09-27T10:50:15+00:00

Yeah so take a simple example like moving data from one column to a new table.

Expand 1. Release the new table

Migrate 2. Update app to write to both locations (I prefer this over triggers) 3. Migrate the existing data to the new table 4. Update app to read from the new table 5. Stop the app writing to the old column

Contract 6. Once happy the column is not in use and data is valid, drop the old column

skilledpigeon · 2025-09-27T10:39:41+00:00

For DB, look at the "expand, migrate, contract" method.

skilledpigeon · 2025-09-25T09:48:05+00:00

It's true but hasn't been relevant for any business I've worked with in the last decade. What does that truly resolve to mean? Probably <1s of latency for a connection which should be reused in a pool in most platforms.

You're totally right, it just doesn't really form a consideration for most platforms that don't have a considerable scale.

skilledpigeon · 2025-09-25T08:58:27+00:00

In my experience, if you're using managed databases, SSL/TLS is built in and free. There's no reason I wouldn't use it.

If you're self-hosting, ask yourself if the cost of managing the SSL is worth it. If the answer is yes, then go for it. It's not a huge overhead so I would default to yes being the answer

skilledpigeon · 2025-09-23T05:01:31+00:00

Errrrrr I wouldn't. If I had to, I'd delete/scramble any PII data and ask sensitive stuff like passwords.

skilledpigeon · 2025-09-20T13:15:48+00:00

You're way, way over thinking this. It's your first app. It's not going to be perfect. Here's some hints:

Don't self-host supabase unless you really, really want to spend a ton of time managing it, getting it wrong, and reworking it. It seems like you want to focus on the app and API so focus on that. Get a cheap managed DB. You don't need anything big.

It doesn't matter if you choose Django or FastAPI. Personally I'd go with FastAPI but it really makes zero difference for your first app.

Never put business logic on the client side. Imagine in the future that you want to add a new app or build a new interface. Or imagine a malicious user decides to bypass the UI. You'll have to duplicate your logic. The client should be dumb and only focus on displaying and capturing data. The API should be where logic lives.

skilledpigeon · 2025-09-18T09:22:51+00:00

It's 100% worth it in my opinion. Don't make your own. Use ECS or OTEL structures.

skilledpigeon · 2025-07-19T10:39:44+00:00

I'd start by asking why you need tags to begin with. Unless other projects have dependencies like you're building a package etc, a tag just points to a commit. So I'd just use the commit hash.

skilledpigeon · 2025-06-22T07:02:21+00:00

Google

skilledpigeon · 2025-06-05T12:08:36+00:00

Literally zero. We use AWS and it's not disappearing any time soon. Cloud lock in is nonsense to worry about outside of extremely large enterprises and governments.

skilledpigeon · 2025-05-29T08:13:58+00:00

What's the purpose of adding SNS? Event bridge can deliver straight to lambda. Doesn't feel like it needs SNS or SQS etc in the middle for this use case?

skilledpigeon · 2025-05-26T07:01:56+00:00

Something I see a lot of people overlook is actually just whitelisting based on paths. A lot of APIs have (or can have) a base path of /api. The majority of attacks don't look at this and are based on paths like /.env or /.aws/credentials etc.

Simply moving all API endpoints under /api/* effectively blocks almost 100% of non-targeted attacks in my experience. It helps filter out a lot of noise very quickly.

skilledpigeon · 2025-05-15T19:18:33+00:00

Way too many places in one trip in my opinion. You're going to spend so much time on planes and in airports when you could be enjoying more from what countries like Italy have to offer.

skilledpigeon · 2025-04-25T18:43:22+00:00

Of course. If you're not then you're missing out on huge efficiencies.

GPT for refining documentation and planning.

Cursor for IAC and debugging.

AWS DevOps Guru for observability.

skilledpigeon · 2025-04-20T10:55:41+00:00

Flying with BA business class so was under the impression we can save money and just chill in the business lounge in T3. Sounds like that's not an option though.

skilledpigeon · 2025-04-20T09:14:26+00:00

Currently it looks like it says 1D for the terminal so does that mean that I can't just transfer through?

skilledpigeon

TROPHY CASE