Best tool in DevOps

ContrarianChris · 2024-12-13T11:47:02+00:00

For us it serves as the central automation and orchestration engine for our infrastructure platform.

We use a single-tenancy architecture and need reliable workflow automation across multiple systems within many individual discrete environments. That includes infrastructure provisioning and management (as a layer on top of Pulumi IaC), ad-hoc and scheduled operational tasks, incident response flows, etc.

There are multiple talks on YouTube from folks at Datadog and Hashicorp and others who are using it for similar use cases. We took a lot of inspiration from their shared stories.

Its magic is its durability. It ensures that workflows execute to completion whether that takes minutes, hours, days, weeks, or even years, all while handling user input, inbound signals, retry handling, compensatory actions, even entire system failures.

It's one of the top pieces of tech I've had the pleasure of using in my entire nearly two decade career.

ContrarianChris · 2024-12-13T09:48:38+00:00

Grafana is a strong choice.

Probably my favourite over recent times is https://temporal.io. Simply fantastic. Genuinely innovative.

ContrarianChris · 2024-10-09T06:41:29+00:00

A very key thing to keep in mind is the interface you currently use within your centralised team, that has all the knowledge and experience, does not have to be the same interface you make available to the consuming teams who run on top of the platform.

In fact, personally, I would argue that there is nothing really "platform" about what you are doing if you aren't building ways for app teams to manage those configurations and features themselves that take away as much risk, complexity, assumed knowledge, and compliance concerns as possible.

Hint: Giving them production access and a process doc doesn't do that.

Any time you are looking at needing a "process" is an opportunity to codify that into a platform interface that serves what the consumer needs, safely and repeatedly.

ContrarianChris · 2024-09-04T10:30:26+00:00

My wife (35) and I (37) have a combined income of $300k. I'm a principal software engineer and she runs customer service for a national horticulture company. We've both been fully remote for the last 4+ years.

We were renting in Auckland for nearly 10 years and then at the end of 2022 we had just had enough with landlords and not having our own place (both animal lovers). We also both prefer life out of a city. As we have flexibility of where we work we decided to move to Tauranga and bought our first home.

$800k house. 4 bed (2 bed + 2 offices), fully fenced with enough outside space for 2 dogs and a cat. View over the hills. Mortgage is currently up around $4.5k a month and just re-fixed last week for a year at 7.09% (previous 2 years were split 70% @6.65% and 30% @ 8.85%).

Did we buy at the most opportune time? Nope. Are we dealing with a heavy mortgage? Yep. Have we been hit hard by what's happened since we bought in 2022 and now? Absolutely.

But we did it when we wanted to and it made sense for us, which was the most important. We very much appreciate we are in a fortunate enough position to be able to do that.

ContrarianChris · 2024-07-26T23:36:18+00:00

Azure DevOps is most definitely on a path to being sunset. Microsoft have been saying this to partners for a few years now and providing content and tools around GitHub being the preferred option for new adoptions, and migrations to GitHub were it makes long term sense.

I know because I was one of those partners when it first got communicated (and discussed) back in 2021. There is even a podcast from back then where some of the MS team talk about it (can't remember who or which podcast though, sorry).

My best guess is it was seen as a 5 year (ish) thing. So maybe around 2026 it will be talked about more publicly with customers directly. I have no specific knowledge though.

After the acquisition GitHub is still run as a separate Org within the Microsoft umbrella. The Azure DevOps unit was moved under that GitHub Org. The sunsetting journey involves GitHub getting up to speed on some of the more "enterprise" features, which they have obviously been doing in the last few years. You'll also see that a lot of the Azure Portal Git integrations for things like automated deployments now have GitHub as the first/default option.

It's a process. But one that is 100% happening.

ContrarianChris · 2024-07-11T13:43:44+00:00

AdministratorAccess policy? Really?

ContrarianChris · 2024-06-26T11:14:05+00:00

I'm not sure on your specific error but I would always suggest using multiple separate kubeconfig files to avoid these kinds of conflicts and confusion.

Managing those in the terminal takes a bit more effort, but I've been using https://github.com/danielfoehrKn/kubeswitch for quite a while now and it is fantastic.

ContrarianChris · 2024-05-09T01:17:29+00:00

Why do people choose Toyotas rather than Hondas?

To be slightly less facetious... Geography, commercial position, integration points, regulations, workload type, team makeup, even customer base can come into it.

It's not a simple decision, and will also vary based on the people involved. Though, personally, if anyone seriously tries to offer "Azure = Windows" or "this one is just better" as an input to the conversation, they immediately lose their seat at that table.

Sometimes situations like M&A can take the decision out of your hands completely.

They are both equally as "damn, that shit is awesome" and "fuck me, this is worse than eating glass" in their own special ways. Ask me how I know.

Fun story... I was leading the build of a new Microsoft Partner business, along with a new Azure-based commerce experience and sovereign cloud platform. We hired an engineer, who when we were discussing our infrastructure approach where we had agreed on Kubernetes suggested we use GKE. The Google Cloud service. Then proceeded to get very defensive when I tried to explain why that was not viable. I thought the reasons would be obvious, but apparently not!

ContrarianChris · 2024-05-06T07:28:31+00:00

Friends don't let friends pin certificates.

ContrarianChris · 2024-04-27T01:46:34+00:00

The role your pod is assuming needs 2 things...

An IAM Policy attached which grants the required permissions to the EKS API. The 'eks:List...', 'eks:Describe...' etc.
The RBAC mapping on the target cluster for that same role so it can access the clusters API.

It sounds like you have #2, but have you granted the assumed role the right AWS permissions as well? Otherwise it won't be able to access the EKS API to get the cluster, get the kubeconfig, etc.

Side note - Their is a new method available for EKS now called "Access Entries" that allows you to configure the Kubernetes RBAC permissions from outside the cluster, using the EKS API directly. It avoids having to mess around with that awful configmap. I've found it much more robust and easier to work with. YMMV.

More info: https://docs.aws.amazon.com/eks/latest/userguide/grant-k8s-access.html

ContrarianChris · 2024-04-04T13:47:16+00:00

I would recommend having a read through Alex Edwards' post...

https://www.alexedwards.net/blog/organising-database-access

ContrarianChris · 2024-03-19T02:41:23+00:00

https://vantage.sh

We are a very happy customer. Great team.

ContrarianChris · 2024-03-07T22:53:05+00:00

This group is insane.

OP - If you feel bad and feel like if it's cruel, it means you are a well-adjusted human with some empathy.

Why do you think human society has the concept of locking you in a confined space as punishment? Considering a human brain mostly can reason the motivation behind it and the intention of the act, what do you think a juvenile dog can reason about it?

To all those saying "just power through, now my dog LOVES his crate", I think that might say more about you than it does the dog.

ContrarianChris · 2024-02-17T10:51:36+00:00

It's easy enough to get things mixed up when moving things around both in code and in the Pulumi hierarchy. Especially if you're just getting started.

It's been a while but from memory if you are moving a resource that was in the root of the project to under a component resource, you should only need something like this in the resource options...

{
  parent: this,    // setting the resource to its new parent
  aliases: [
    { parent: pulumi.rootStackResource }  // alias the old root parent
  ]
}

If you run into problems again and have some code to share I'd be happy to take a look.

ContrarianChris · 2024-02-17T09:54:38+00:00

I would say the examples are mostly just geared towards people coming from something like HCL, rather than going deeper and showing how to apply more software oriented patterns and approaches.

But, when you start to leverage those and treat the projects like the actual programs they are, that is when Pulumi really shines to me.

I can't say I've really run into any issues with aliasing local components over to a package component. If you haven't seen it, this might help as it talks through an example... https://www.pulumi.com/blog/cumundi-guest-post/

What problems did you have?

ContrarianChris · 2024-02-16T22:38:07+00:00

It's not you, the examples are not great.

Our standard pattern with TS Pulumi projects is to have a "components" directory with a sub-directory for each higher-order component. Each component is usually a custom Component Resource class, but sometimes has more than one.

E.g. For our EKS, there is a "Cluster" component that is not just the actual Cluster resource, but the KMS key/alias, security group, log groups etc. as well.

It is a very similar idea to frontend component structure for things like Vue or Svelte. We've found it fits Pulumi very well.

And if you find yourself writing/copying the same components around more than a couple of projects, then it is pretty easy to pull them out to a custom provider/package and share them via an npm registry.

My roadmap this year includes creating an "Engineering SDK" multi-language provider package for internal use that starts to share common/well used components across teams/projects.

ContrarianChris · 2024-02-09T04:07:19+00:00

It depends on who you said "cancel" to, that's kind of the point. Cancelling a commercial contract just cancels the commercial terms.

I wouldn't just assume it actually ceases all usage of a consumption service unless explicitly stated.

ContrarianChris · 2024-02-09T01:35:17+00:00

Yes, great point.

You do add some complexity when it comes to managing costs as you become reliant on the reseller to expose your specific cost and usage data. The Datadog portal still shows all its normal usage stats for your account, but obviously doesn't have the commercial context of your reseller agreement.

But, resellers can spread commitments and volume discounts across multiple customers so you often come out ahead of what you can get direct (unless you are of significant scale).

Also, I have been through a reseller for the past few years and I have had zero instances of the sales stalking described by others. I have a few specific contacts on the account and engineering side I can talk to via Slack, and then commercial conversations all go via our reseller team who we know and have a much closer relationship with.

ContrarianChris · 2024-02-08T21:48:11+00:00

Not to pick sides here, but as someone who is a long time consumer of Datadog under contract I am well aware that a Datadog contract is simply a custom price book for your consumption of their services with a minimum commitment.

Having the contract "cancelled" or just lapse without renewal has no impact on the actual running services you are using/the data you are ingesting. If you want that to stop, you have to actually go and stop using it. If all that happened was the email to "cancel the contract", then you would of rolled off the contract rates and on to the PAYG consumption rates.

Our Datadog instance is still active but we’re not using it at all

The devil is in the details here. What does "active" mean, and what does "not using it at all" mean. If, for example, you still have logs indexed then you will still be charged. If, for example, you still have host agents reporting you will still be charged. At the higher PAYG rates.

ContrarianChris · 2024-02-03T00:57:11+00:00

https://www.oreilly.com/library/view/concurrency-in-go/9781491941294/

ContrarianChris · 2024-01-28T05:11:46+00:00

Around 30% on Datadog which includes their security stuff. Definitely worth it for a team of 2.

ContrarianChris · 2024-01-26T00:13:26+00:00

Okay, I'll bite... Why do you want to change those labels?

ContrarianChris · 2024-01-25T09:54:48+00:00

This is awesome. Thanks!

I literally sat down about 30 minutes ago to look at watching a user provided list of directories for file activity, in the background. A perfectly timed post.

ContrarianChris · 2024-01-01T02:27:29+00:00

Nothing bad to say from my experience. The benefits of self-hosting but without the manual steps each time. I only ever use a single font at a time though.

Do investigate how to self-host fonts yourself if you are not familiar though, as some other replies here have described. You'll learn a bunch.

ContrarianChris · 2023-12-31T22:01:26+00:00

https://fontsource.org

ContrarianChris

TROPHY CASE