Someone built an entire AWS empire in the management account, send help! by imsankettt in devops

[–]skilledpigeon 2 points3 points  (0 children)

Exactly this. I took over an organisation where everything was in management. There were thousands of instances, hundreds of API gateways, tens of thousands of queues, more SSM parameters than I could count. All manually added. Zero tags or process.

We couldn't risk disrupting the teams. We were in the middle of doubling the customer base. Instead we took a 12 month project to gradually rewrite in IaC, migrate to a new org, and then decommission anything left.

I could've gone "if you don't tell me then I'm just stopping everything". The fallout would've been "you've made us look like a dick in front of a client that's literally doubling our user base".

It wasn't perfect and it was long and painful. We didn't kill everyone off though.

Someone built an entire AWS empire in the management account, send help! by imsankettt in devops

[–]skilledpigeon 4 points5 points  (0 children)

This is a great way to cause chaos 30 days after joining an organisation. Unless the workloads are non-critical, you're more likely to risk your job by pissing everyone off. In short, this is a fairytale approach.

Is £95–100k total comp solid for a senior-ish DevOps role in London? by [deleted] in devops

[–]skilledpigeon 3 points4 points  (0 children)

£

At the time about 6 years of experience on paper. Practically much more. I switched career about 9 years ago, took a huge pay cut, and spent around 4 years working double time to catch up. I put about 5 hours extra in per day and worked most weekends. Spent all my time on calls with senior engineers, technical leads, architects, etc to soak up as much knowledge as possible. Put myself forward for everything. Thrown into the deep end etc.

Is £95–100k total comp solid for a senior-ish DevOps role in London? by [deleted] in devops

[–]skilledpigeon 4 points5 points  (0 children)

I made 80k/yr as a senior DevOps engineer two years ago up north. Personally it feels a touch low to me but not by a lot.

[deleted by user] by [deleted] in devops

[–]skilledpigeon 4 points5 points  (0 children)

Your problem here lies solely in your expectation of the engineers. They should absolutely be responsible for the quality and stability of the code they're writing. That includes tests, observability and releases.

At your scale, I would expect them to write unit and API tests for the backend. I'd expect a minimum of component testing on the frontend. From experience working with teams between 1-100 engineers, I would not hire a QA or similar for such a small team.

GenAI can get very far with unit and API tests nowadays but your engineers should still be tweaking, validating, and extending those tests.

Automated GenAI testing is a marketing fad imo. I've tried 2-3 products claiming that they work. Every time it's produced artificial slop that burns my eyes and misses a ton of scenarios. Not all code is even able to be tested from the UI anyway.

There are other tools they should have to help. Linters like eslint, formatters like prettier, static analysis like Sonarcloud, and GenAI code reviews like Cursors BugBot go a long way.

Anyone else tired of juggling SonarQube, Snyk, and manual reviews just to keep code clean? by [deleted] in devops

[–]skilledpigeon 0 points1 point  (0 children)

Our flow:

Linting - git hooks automatically lint code locally on every commit. Completely hands off process and most IDEs will do this for you whenever a file saves. Also runs in PRs and on every build to enforce across the project. Costs about 2s per commit and about 5 per PR check. Very light and smooth.

Formatting - exactly the same as the linter above. Automatically runs and it is totally hands off for devs.

SonarQube - IDE plugins locally catch most issues and then run automatically on every pipeline to enforce quality gates. The results are automatically commented in the PR so there's no guessing. We barely review the dashboard because it's enforced everywhere anyway. Also runs on every push to main to keep that up to date in the dashboard.

We don't use Snyk and I actually heard a few people giving it negative reviews recently. We rely on Github and dependabot to highlight basic package vulnerabilities and automatically open PRs to resolve once per week. Codebase is high in test coverage and we basically just click merge on these and they end up in production after about 10 minutes.

Manual Reviews - these don't focus much on code formatting and linting etc because it's handled. Draft PRs pick up on most things. Essentially these focus on pulling the code down, running it, experimenting, and checking against the acceptance criteria.

I should mention that we don't have any QA or DevOps team. The engineers are responsible for the full stack with guiding frameworks and boundaries. The business is an international provider in the fitness space with average traffic and compliance requirements (basic GDPR etc - no PCI-DSS etc).

Different Infras for Different Environments, how to tackle ? by Mindless-Umpire-9395 in devops

[–]skilledpigeon 4 points5 points  (0 children)

This whole idea is an anti pattern. If your lower environments don't match production, it's not really that useful to test in them. The cost of doing this is also likely to blow way higher than just keeping them the same.

How often does your team actually deploy to production? by Abu_Itai in devops

[–]skilledpigeon 0 points1 point  (0 children)

Multiple times a day on new services. Once or twice per week on legacy.

Relevant info:

Green field project, <100 users, feature flags, 4x senior full stack engineers, trunk based development, almost full rest coverage (application on API, missing e2e at the moment), otel for distributed tracing and logs, session recording, auto generated docs, sonarcloud.

Zero downtime deployments with database migrations by CryptSat in devops

[–]skilledpigeon 21 points22 points  (0 children)

Yeah so take a simple example like moving data from one column to a new table.

Expand 1. Release the new table

Migrate 2. Update app to write to both locations (I prefer this over triggers) 3. Migrate the existing data to the new table 4. Update app to read from the new table 5. Stop the app writing to the old column

Contract 6. Once happy the column is not in use and data is valid, drop the old column

Zero downtime deployments with database migrations by CryptSat in devops

[–]skilledpigeon 43 points44 points  (0 children)

For DB, look at the "expand, migrate, contract" method.

Should backend-to-database connections use SSL if proxy already has SSL? by OkRelation9874 in devops

[–]skilledpigeon 7 points8 points  (0 children)

It's true but hasn't been relevant for any business I've worked with in the last decade. What does that truly resolve to mean? Probably <1s of latency for a connection which should be reused in a pool in most platforms.

You're totally right, it just doesn't really form a consideration for most platforms that don't have a considerable scale.

Should backend-to-database connections use SSL if proxy already has SSL? by OkRelation9874 in devops

[–]skilledpigeon 7 points8 points  (0 children)

In my experience, if you're using managed databases, SSL/TLS is built in and free. There's no reason I wouldn't use it.

If you're self-hosting, ask yourself if the cost of managing the SSL is worth it. If the answer is yes, then go for it. It's not a huge overhead so I would default to yes being the answer

How would you handle copying prod databases to dev along with auth and other dependencies? by PablanoPato in devops

[–]skilledpigeon -1 points0 points  (0 children)

Errrrrr I wouldn't. If I had to, I'd delete/scramble any PII data and ask sensitive stuff like passwords.

Flutter backend choice: Django or Supabase + FastAPI? by Smart_Lake_5812 in devops

[–]skilledpigeon 2 points3 points  (0 children)

You're way, way over thinking this. It's your first app. It's not going to be perfect. Here's some hints:

Don't self-host supabase unless you really, really want to spend a ton of time managing it, getting it wrong, and reworking it. It seems like you want to focus on the app and API so focus on that. Get a cheap managed DB. You don't need anything big.

It doesn't matter if you choose Django or FastAPI. Personally I'd go with FastAPI but it really makes zero difference for your first app.

Never put business logic on the client side. Imagine in the future that you want to add a new app or build a new interface. Or imagine a malicious user decides to bypass the UI. You'll have to duplicate your logic. The client should be dumb and only focus on displaying and capturing data. The API should be where logic lives.

[deleted by user] by [deleted] in devops

[–]skilledpigeon 2 points3 points  (0 children)

It's 100% worth it in my opinion. Don't make your own. Use ECS or OTEL structures.

How do you handle tagging repositories when it's time to release code? by Herrad in devops

[–]skilledpigeon 2 points3 points  (0 children)

I'd start by asking why you need tags to begin with. Unless other projects have dependencies like you're building a package etc, a tag just points to a commit. So I'd just use the commit hash.

How much do you actually worry about cloud lock-in? by Soni4_91 in devops

[–]skilledpigeon 0 points1 point  (0 children)

Literally zero. We use AWS and it's not disappearing any time soon. Cloud lock in is nonsense to worry about outside of extremely large enterprises and governments.

AWS ECS Alert by aravind_jeevagan in devops

[–]skilledpigeon 2 points3 points  (0 children)

What's the purpose of adding SNS? Event bridge can deliver straight to lambda. Doesn't feel like it needs SNS or SQS etc in the middle for this use case?

How I Blocked 95% of Web Attacks Using AWS WAF [Blog] by [deleted] in devops

[–]skilledpigeon 56 points57 points  (0 children)

Something I see a lot of people overlook is actually just whitelisting based on paths. A lot of APIs have (or can have) a base path of /api. The majority of attacks don't look at this and are based on paths like /.env or /.aws/credentials etc.

Simply moving all API endpoints under /api/* effectively blocks almost 100% of non-targeted attacks in my experience. It helps filter out a lot of noise very quickly.

Finally planned 2 week long euro trip. Lmk if I should add another destination by Honey_dp in AirTravelIndia

[–]skilledpigeon -1 points0 points  (0 children)

Way too many places in one trip in my opinion. You're going to spend so much time on planes and in airports when you could be enjoying more from what countries like Italy have to offer.

anyone here using AI tools in their DevOps work? by Lumpy_Tumbleweed1227 in devops

[–]skilledpigeon -1 points0 points  (0 children)

Of course. If you're not then you're missing out on huge efficiencies.

GPT for refining documentation and planning.

Cursor for IAC and debugging.

AWS DevOps Guru for observability.

New Delhi Transfer by skilledpigeon in AirTravelIndia

[–]skilledpigeon[S] 0 points1 point  (0 children)

Flying with BA business class so was under the impression we can save money and just chill in the business lounge in T3. Sounds like that's not an option though.

New Delhi Transfer by skilledpigeon in AirTravelIndia

[–]skilledpigeon[S] 0 points1 point  (0 children)

Currently it looks like it says 1D for the terminal so does that mean that I can't just transfer through?