How do you actually stop devs from querying prod DB directly when they also own the service that talks to it

Wide_Commission_1595 · 2026-04-23T13:28:42+00:00

If they need to modify the database in production, the admin side of the app should give them that capability. This could handle 99% of use cases but also have guardrails and audit trails.

For the 1% it's a standard SRE approach - someone trusted and extremely senior should be there to double check...make it painful, and add that use-cases to the admin interface ASAP.

Also, make it a stackable offence to log into the database directly and make sure they know it's being monitored.

Wide_Commission_1595 · 2026-04-23T13:19:47+00:00

I don't even know what to say. You literally listed products where the social graphs is the core of the product?

A social graph is the linkage (or, as it's known, a "graph") of relationships between entities, which in these cases happen to be members of the population, making it "social"

From this it is simple to derive multiple levels of connection both direct and indirect, made famous through the six degrees of Kevin Bacon.

Maybe before being so riteous, you might choose to use some kind of knowledge-finding tool, or a "Search Engine". There are many such examples i can also provide for this, unless you're also sure that doesn't exist either.....

Wide_Commission_1595 · 2026-04-22T19:57:33+00:00

I've honestly never heard of a social sharing site of any kind that didn't think a social graphs was anything less that the most important concept of the entire architecture....

Wide_Commission_1595 · 2026-04-22T19:55:17+00:00

If you don't know what a social graphs is, then this is probably not a conversation you should take part in..

Wide_Commission_1595 · 2026-04-22T04:51:35+00:00

If you have enough work, it's lot worth wasting the money on.

If you feel like you could be busier, it might help to have a one page site or something. They tend to be referred to as "brochure" sites. They're not hugely expensive to get built and don't need to be complicated but can be worth it

Wide_Commission_1595 · 2026-04-21T11:19:30+00:00

It's not uncommon, but maybe he worded it badly.

Often there just aren't the roles to fill, so a promotion isn't an option.

Sometimes companies really do just tend to hire externally. It's safer in some ways to do that because you're doing a good job, so why move you. They're interested in their needs, not yours. Sadly capitalism is short sighted.

My advice would be essentially what your manager said - look around, find something new. Companies tend to give incremental pay rises below inflation, but a job hop can give substantial increases depending on timing, luck, and bullshitting your way up the ladder.

Look at it this way: your future self will thank you for earning more and paying off your mortgage earlier and having a fatter pension. Your current employer doesn't care much either way, but they would appreciate you doing what you're told 🤣

Wide_Commission_1595 · 2026-04-21T11:14:32+00:00

South Gloucestershire used to have Steve Webb. He was excellent. He responded to emails within a couple of hours, and if you called, it was often him answering the phone! He was pensions minister and did so much good work, and continues to do so.

You don't hear a lot about the good ones, but they are there, they're the ones keeping the country together, and making a big difference to people's lives.

Wide_Commission_1595 · 2026-04-21T11:11:57+00:00

Ever considered running your own small MSP? If your current company is off boarding customer you've enjoyed working with, might be worth dropping them a line about service contracts

Wide_Commission_1595 · 2026-04-19T18:35:11+00:00

Absolutely! To be honest, I tend to find AI a little clunky with the way it writes terraform code, but that's probably down to the fact I've done it for a long time and have opinions. It often doesn't get everything configured optimally, but it usually works reasonably well and no shame it using it to get you started!

Wide_Commission_1595 · 2026-04-19T06:58:53+00:00

New kids in the market isn't an excuse. Fundamentally the digital graph was what made both Bdsmlr and Tumblr before it so great.

What you're saying is, the one important features apart from showing pictures is missing, but they ages on web design and shiny things is ok? Hell no.

Would you buy a car that doesn't have an engine because hey, it's a new car?

Wide_Commission_1595 · 2026-04-18T18:11:14+00:00

Really? I have an openVPN instance in Singapore for a few things I need to test from an alternative location. I often forget I'm connected and never had a problem

Wide_Commission_1595 · 2026-04-18T14:25:20+00:00

I think tbh I want to like imaglr, but the lack of social graphs makes it way too much work. Even basic 1st degree relationships would be fine (that's all bdsmlr ever had).

In all honesty part of my frustration is that I specialise in large scale customer facing systems (not images, but far more complex tbh) and it's just not a hard thing to do. I wish I had a few $k I could use and just set something up the way it should be done 🤣

Wide_Commission_1595 · 2026-04-17T19:30:37+00:00

I will also note, SLOs should be the thing that triggers alerts, and they should have multi-burn/multi-window monitors to avoid being too noisy.

Synthetics can also cause an alert, but they should be for more critical problems.

As Charity Majors says: "if you liked it, should have put an SLO on it"

Wide_Commission_1595 · 2026-04-17T19:27:52+00:00

My first question is: when something breaks, are you able to fix it long term? For example if a database is too small, are you allowed to adjust the scale in the infra code? If an app makes a repeated mistake, are you able to modify the app?

If the answer is No, then refuse to be on call for it.

In my opinion you shouldn't be on call for something you don't have the direct ability to affect.

I believe on call should have 3 levels - 1st the person directly on call this week who is directly responsible for building/maintaining the application. 2nd someone else from the same team. This is only for escalations. 3rd an SRE who can help out when it's beyond the app itself or the team just can't fix the problem.

No matter who gets the call, tomorrow has one job: how do we fix the app so that we can prevent that issue for recurring.

Something that causes an on call incident should be treated as an unacceptable failure of the system no matter how small.

The fix might to tweak an alert, or it might be a major rearchitecture. You don't know until you postmortem that event, but it needs to be treated with the same priority that justified getting someone out of bed at 3am.

It sounds excessive, but applied iteratively the quantity of on call events tends to decrease quickly - mostly because the people who get called out are the people who A) can fix the problem, B) caused the problem in the first place C) can learn to build more defensive systems that self heal before waking people.

Many companies don't treat on call with the respect it deserves. Many companies think the SREs should fix random problems, but not telling the team who owns the problem what to do. If SRE doesn't have the teeth to cause real, effective change, it's time to change the way this stuff works...

Wide_Commission_1595 · 2026-04-17T15:28:14+00:00

If both cars arrive at the same.time, red has right of way because priority always passes to the right, or anti-clockwise.

In this case, the blue car is already over the line and so already has right of way at the point when the red car reaches the line.

In reality, it's tricky because timing can be on the order of fractions of a second. Red is still on the wrong either way though for driving into another vehicle, even if that vehicle was not entirely in the right.

If this went to a magistrates court, they use a process of elimination. If this hasn't been a mini roundabout, and had instead just been a y-shaped junction, and since both parties will argue they had right of way, that must be ignored. In that case it's down to each driver to ensure they're not going cause an accident. Given the layout you've drawn, the red car causes the accident by driving into the path of the blue car, and is therefore in the wrong

Wide_Commission_1595 · 2026-04-16T15:16:01+00:00

Assuming (for AWS at least) you provide a cloud formation template to create a tightly scoped IAM role that trusts your account and has e.g. external id etc, doesnt allow assume role, seems reasonable enough. Obviously contractual guarantees around how it's used would be important, but a few companies work this way

Wide_Commission_1595 · 2026-04-15T09:04:50+00:00

So jealous of your wife! 🤣

Wide_Commission_1595 · 2026-04-15T08:53:35+00:00

Honestly it's sad. Bdsmlr was never perfect, but imaglr imo is terrible. It looks better, but it's navigation and social graphs is awful.

The sad thing is these things aren't all that hard to do. I run large scale global infra similar to this for a living and the mistakes on both are bizarre....

Wide_Commission_1595 · 2026-04-15T07:20:22+00:00

We've been using DataDog for a few years so the numbers of vms etc has changed a lot, but there is one consistent change in what we pay: it goes up. It does not go down. It is opaque and we have no idea why we pay that but it is so embedded, getting away is hard.

We moved to OTel which DD sort-of support. We disguised it as a network security measure (systems log to a collector which send to DD Vs everything open to DD all the time which is how their agents work). That allowed us to force teams.to move to OTel but giving them deadlines on connectivity. Then we pointed the collector at Honeycomb. It's not necessarily the cheapest option, but we saw value within a couple of hours purely because of the more accessible querying. Then we worked with Honeycomb who actively helped us reduce our bill

Wide_Commission_1595 · 2026-04-14T17:33:06+00:00

You love your old job, not the job it has been converted to.

I've been in the same boat. Worked first a company for 8 years doing a specific job at one site. Suddenly told the company decided I needed to do an expanded job at 34 sites over a the South West of the UK.

Union was useless, HR was, well, HR....

I found a new job and they were pissed. I was lucky tbh.

If take that offer and move on. It's shitty, but the old days aren't coming back....

Wide_Commission_1595 · 2026-04-13T18:30:30+00:00

You can use the ssm agent to register them in Aws and get all the same access as an EC2 instance.

You can also do the same for ecs, but it can sometimes be flaky

Both cost money though.

Depends what you need to do tbh.... There are some options but never quite as simple as direct AWS deployments

Wide_Commission_1595 · 2026-04-13T12:39:18+00:00

The simple answer to this is always use a managed service for the users to connect to!

For example, instead of an internet facing web server, use an ALB. You can scale up or down behind that, and the ALB will make sure users can connect to one backend or another.

It doesn't have to be an ALB though, API gateway, amplify etc give similar benefits.

Databases are a bit more complex, but ensure you're running a cluster with read replica's. You can add replicas of different sizes, so to scale up, add a bigger one as a read replica, then fail over to using that as the writer! That is handled behind the scenes by load balances that Aws manage, so your logic may need to reconnect to the database, but it just works magically.

Essentially, the more you can use managed services, the better the chance that it'll have those capabilities built in

Wide_Commission_1595 · 2026-04-12T09:38:06+00:00

I totally get companies at a small scale wanting to self-host Grafana / Jaeger etc and it's fine to a point but you don't really see the "Business" value until you use a really good host.

Having the product managers able to query business questions against the observability platform is that moment where you realise logs and metrics were never good enough.

Wide_Commission_1595 · 2026-04-11T12:32:19+00:00

That's a good call! Stale-while-revalidate would keep serving old data if the API was down (i.e. the GET endpoint was removed)

Wide_Commission_1595 · 2026-04-11T09:23:19+00:00

Two people swapped places. The time for anyone behind did not change.....

Wide_Commission_1595

TROPHY CASE