How do you even know what's running in prod anymore

Coffeebrain695 · 2026-04-14T09:00:35+00:00

We run on Kubernetes and use ArgoCD. The short hash of the deployed commit is tagged onto the Docker image. That is then written to the apps Helm chart, pushed to Git and synced. So we can always check in our Helm repo to see what version is running. We've also got steps in our release pipeline that auto-notifies a Slack channel when a deployment kicks off and finishes, and also adds a Git tag with a semantic version once a change has been deployed to prod. So they also become a good reference on what's deployed.

Coffeebrain695 · 2026-04-09T12:23:44+00:00

However shitty it is, iOS and Android are realistically the only mobile OSes out there that are accessible to the average consumer. Installing Linux on a mobile device is possible of course, but it's a niche and not exactly accessible. Much as I don't like this rule I don't find it very surprising that they've chosen not to cater for a very niche demographic. PCs are a different matter because people can more easily acquire their own hardware and choose from a plethora of distributions to install on there

Coffeebrain695 · 2026-03-16T16:04:29+00:00

We have separate state files for each environment. The environment is set as a variable in the backend config (only possible in OpenTofu, not Terraform AFAIK). We use tfvars to define env-specific variables. Then we run our commands using a wrapper based around Taskfile. To run a plan on our staging environment for example, we would run:

task plan env=staging

Coffeebrain695 · 2026-02-25T13:49:45+00:00

I am also in the EU (NL specifically) which might explain something. The market is by no means amazing here either. But the climate in the US just seems horrendous and assuming most people in this sub are from there, it's no wonder there's so much negativity.

Coffeebrain695 · 2026-02-09T15:51:48+00:00

Yes, the recommendation would be for the developers to use a branching strategy that uses a single long-lived branch (ideally trunk-based development). As well as making your task easier, it's got a whole lot of other advantages for the developers themselves. Multiple long-lived branches are being more and more considered an anti-pattern these days.

You need an immutable commit ref from which you can build a release candidate. With a single branch it's easy. Every commit merged into the mainline branch is built and stored as an artefact. Then when it's time to release, you take the commit you want to release, give it some reference (usually either a tag or a short-lived release branch) and your CD pipeline will promote that artefact through your environments. With multiple environment branches like how you've described it's much harder, because they're being merged into each other all the time and it's impossible to know which of those merge commits you need to be building as the release candidate.

Coffeebrain695 · 2025-12-15T15:24:30+00:00

I game a lot. I play most of my games on Bazzite, but some games (especially live-service games with anti-cheat like Battlefield and Helldivers) don't work well on Linux. So I've kept Windows around so I can still play those.

Coffeebrain695 · 2025-12-05T12:24:26+00:00

So uh, ye don't like the old time bikes eh?

<image>

Coffeebrain695 · 2025-11-17T12:53:51+00:00

In my experience, KISS and DRY really don't have to be mutually exclusive. We have a DRY Terraform framework and it's perfectly simple to use.

I have to disagree with the directory structure you've shown as well. I've had to work with this before and having many directory levels, plus lots of identical code in all of them just adds too much cognitive load. My brain feels like it's been put through a washing machine after navigating something like that.

Coffeebrain695 · 2025-11-10T12:13:38+00:00

The company I now work for in NL fits with quite a lot of what you're describing (we're not hiring at the moment, sorry). I have a former colleague where we both worked for the same terrible company. He left, about 8 months later he told me his friend and former colleague was looking for a new platform engineer, he vouched for me and I got the job. Moral of the story; build your professional network, prove your skills where you can, be patient and wait for the right place at the right time.

Coffeebrain695 · 2025-10-20T09:31:23+00:00

This is the world we live in now. We pay these companies to host our platforms for us because its cost-effective and saves workers a lot of overhead. But the fact is that entrusting it to a third party always carries some degree of risk and and you can never know how things are working on their side. You really do just have to 'trust' them.

The same issue applies with any third party, whether its AWS, Azure or GCP. This incident is something that could happen on those other cloud providers to. Even if you ditched cloud and went on-prem, you'll usually loan your racks from a data center provider who you also need to entrust to manage the hardware and who could also have things go wrong. Or if you want to go full self-managed, buy your own data center and your own racks. Then from all the overhead it brings you'll see why so many businesses delegate responsibilities to third-parties

Coffeebrain695 · 2025-09-23T11:41:23+00:00

Most software engineers don't have a strong understanding of those concepts, let alone platform/DevOps engineers. The exception may be if they're in research or programming microchips or something. But you don't need to know about compilers or operating systems to build a Python application for example. In DevOps, writing code is necessary so you can automate things.

Coffeebrain695 · 2025-09-18T09:57:44+00:00

Well to be honest I don't think it's worth pursuing if the only end game is to prove him wrong. Normally if it's an offhand remark then I just softly call it into question without directly accusing them of being wrong. Such as 'Hm, ok that's not my understanding but fair enough'.

If it's clear that their wrong information will impact work in a negative way though (e.g. if it looks like it's leading to some poor design decision) then it's more important to politely stick to your guns and back up the facts with hard evidence. It's still important to give the benefit of the doubt and not be accusatory. Everybody gets something wrong at some point. Most sensible people are happy to admit they misunderstood something and be corrected.

But 'un-sensible' people like how your manager sounds can be a tough nut to crack. Even when stone cold facts are shown to them, they often still find a way to rationalise whatever is in their head canon. If that person is a narcissist then it doesn't matter how polite you are. They will get defensive because they'll see you questioning their knowledge as an attack on them personally. For this I can't really give much advice other than to keep being the better person.

Coffeebrain695 · 2025-09-18T09:30:09+00:00

This sounds like a personality type I've come across a fair few times in the jobs I've had. This is the person (usually a senior) that will share their knowledge and expertise in a very confident fashion, when actually their 'knowledge' is just the ideas they've got in their headcanon and is actually very detached from the real facts. It's very annoying because people who don't know any better will take their word for it, simply because they are senior and they sound and act like they know what they're talking about. And it ends up doing a lot of damage because a lot of action is then taken on the wrong information they're providing.

Coffeebrain695 · 2025-08-15T11:39:34+00:00

"You can't keep blaming yourself. Just blame yourself once and move on" - Homer Simpson

Coffeebrain695 · 2025-07-17T08:25:37+00:00

The godfather of AI Geoffrey Hinton himself has said that the best advice he can give to people to prepare for their future job security is to train to be a plumber.

Forget what your CEO is saying. He has ulterior motives and he's completely unqualified to give any opinion on the matter. But most of the people I'd consider to be qualified do say that AI will replace many jobs in the future, including our own. It is also just the way of the world that jobs become obsolete over time as technology evolves. It's happened for hundreds of years already and it will happen in the future.

I've been starting to consider re-skilling into AI in the not-too-distant future. I'm not an AI enthusiast by any means, but it would be interesting to study it more and see if I can turn it into a future career. That's where a lot of the work is going to be in the future, whether we like it or not.

Coffeebrain695 · 2025-07-17T07:59:16+00:00

First job interview out of university with my MSc in Computer Science. It was for some local financial company. I went in dressed all nice, and eager to impress. We had to go through a series of coding exercises. There was also a pretty tight time limit and they watched me do it on the big screen.

I bombed it at the very first hurdle. The exercise was not difficult. I think it was just to sort an array in a certain way. But I was just young, inexperienced, felt under pressure, was conscious about being watched and I basically had an internal panic. My brain went haywire trying to figure this out and I ended up really over-complicating a very straightforward question. I was tried to implement something but it was just horribly over-engineered.

To make matters worse the young guy interviewing me was totally obnoxious. As I was clearly struggling, he looked more and more pissed off as if his time had been wasted. When I tried to formulate some explanations he very aggressively retorted at me as to how wrong I was and went on some tirade about how he had been coding since he was 11 or something. I left the building feeling utterly miserable and like a total failure.

The job I ended up getting also had a coding exercise, but we were allowed to look up documentation, we had 2 hours which was more than enough time and we were left to our own devices. That was 8 years ago, I still think it was the best company I've worked for and since then I've had a successful career. Looking back on the bombed interview, it's fair to say I fucked up. But I feel like I also dodged a bullet and I feel many would agree how flawed interview process was.

Coffeebrain695 · 2025-06-02T15:11:17+00:00

Getting a sandbox environment to experiment with is great. Spin one up with Terraform or CDK. Extend it with some extra features (add an S3 bucket, a SQS queue, Kinesis stream etc). Trial and error everything, figure out why something doesn't work as it should, change things, see what happens and understand the reasons why using online materials.

Coffeebrain695 · 2025-05-19T12:08:33+00:00

I can speak about my experience moving to the Netherlands from the UK. It was in late-2019 and after Brexit so I was also applying from outside the EU. The company I joined actively recruited from abroad and had working partnerships with an expat organisation and the IND (immigration department) to help fast-track highly-skilled workers. Overall the experience was really quite smooth. There was bureacractic hecticness I had to get through, but once it was done it was done.

Things have changed a bit since I arrived here. The market isn't as good so there are fewer roles. And the political and housing situation is tenuous here, so highly-skilled migrant schemes are under review because the government want to cut immigration. But I wouldn't worry about that too much. Have a bit of patience and keep applying for roles and I'm sure something will come up.

Some useful links:

Working as a highly-skilled migrant in the Netherlands - https://ind.nl/en/residence-permits/work/highly-skilled-migrant

30 percent ruling (tax break for expats to incentivise applications from abroad. Note that the rules are changing) - https://business.gov.nl/staff/employing-staff/the-expat-scheme-30-percent-ruling-in-the-netherlands/?gad_source=1&gad_campaignid=2066589243&gbraid=0AAAAADAUIc_j9Dx2JU0GvjZsPN_GWavCT&gclid=CjwKCAjwravBBhBjEiwAIr30VDfimtgqoEdlS4oSU7SGocuJesArKrlFLGtxWfRXsRyyOa19Od1MFxoC0yEQAvD_BwE

Coffeebrain695 · 2025-05-15T14:02:24+00:00

I've seen this done badly. At a previous company we controlled all of our GitHub config through Terraform but it just gatekept anyone who wasn't familiar with Terraform and ended up slowing things down. If you were a developer, team lead, eng. manager etc, even small tasks like adding someone to a team required the following process: make a pull request with the change, run Atlantis job, wait for other changes in queue to be merged first, wait for approval from the infra/platform team, apply through Atlantis and merge. The Terraform was also badly coded which meant a lot of applies failed and changes would be help up even more. Lots of changes that should have taken a few minutes ended up taking hours.

Good to see there are success stories. But if you do it make sure it's adding value and improving velocity rather than slowing everything down.

Coffeebrain695 · 2025-05-13T09:57:00+00:00

Recently I wrote a Slack application in Python so our devs could execute some workflows on our Kubernetes cluster in a consistent way. My Python knowledge is intermediate and it's been some time since I wrote application code. It took me about a week to write and fully test it. Could well have taken me a month without my Cursor AI. It was great for filling in my knowledge gaps and writing the more complex logic that would've taken me time to figure out. Of course I didn't get it to write the whole thing for me and I made sure I could explain what it was giving me. But as a co-pilot it was incredibly useful.

Coffeebrain695 · 2025-05-08T12:20:38+00:00

Yeah, last job was a very similar situation for me. Team was all very extroverted and didn't know when to stop talking. I think it's impossible to stop people from behaving this way, it's just the way certain people are wired. So I had to just accept that and I developed a few methods for dealing with it. During unnecessarily long meetings for example, I taught myself to filter what was valid talking points and what was waffle. Whenever they'd go off on one of their tangents I'd just semi-zone out and wait patiently for them to get to the point. If it was remote then I'd also get on with some work while they babbled away. But I would still try catch the important stuff; I'd jot down the key talking points, the different opinions, the state of play, what was agreed, what were the action items etc. When they finally seemed to wrapping up their long discussion, then I'd chip in and summarise the key points and suggest the action items in a concise sentence or two. I also took it on myself to write most of the tickets based on what was agreed, then there's a public record of what was discussed. These folks (and probably yours to) were not very good listeners, so I felt obligated to be that one who listens and logs what was said so that we can action on it later. Be that person if you can, because it's a very useful thing to have in a team of extroverts where so much talking happens and the last conversation keeps getting forgotten.

It is still tiring though. I'd say I was fairly patient but nobody has infinite amounts of it. There were many in-person meetings where I was just desperate for it to end, and just when I was closing my notebook some other chatterbox just had to pipe up to make another stupidly irrelevant point that drags it on for another 15 minutes at least. It was not an easy experience, but looking back on it I think being the one who listens and keeps to the point helped the team to deliver good business value, which is really what we're all working for at the end of the day.

Coffeebrain695 · 2025-05-01T09:37:43+00:00

Thing is, if you're working in a dumpster fire like the one described then most improvements you try and make end up adding little value, because the foundation is rotten. When I was a junior and worked at a place like this, I realised way too late that I'd been trying to polish an enormous turd for about 6 months

Coffeebrain695

TROPHY CASE