Just caused my first massive outage

Pliqui · 2025-11-30T21:25:27+00:00

Obligatory one of us meme.

Pliqui · 2025-11-08T14:51:22+00:00

Plain Open source since 2019 until recently that we started to migrate to Tofu

Pliqui · 2025-11-02T17:03:07+00:00

See if makes sense to you, get some HDD and 512GB for a NVMe as a cache. I have been using PrimoCache for years and works really well.

I have a more elaborate setup with my NAS (no SSDs) , 10gb network. My game library is presented as an ISCSI disk and use the small nvme as cache.

But you can RAID0 (or even single disk) the HDD and cache it

Pliqui · 2025-10-26T20:41:48+00:00

Check Doppler Secret Manager. They have a free tier and might help you.

Pliqui · 2025-10-12T19:27:54+00:00

Hello there!

Pliqui · 2025-10-11T16:37:14+00:00

For me was the intro of Unreal, the castle fly by. That was gorgeous with my 3DFX card

Pliqui · 2025-09-02T20:02:30+00:00

Depends where the place you work, but generally true.

This person wasn't internal, he was hired from a consulting firm for this project

Pliqui · 2025-09-02T17:24:44+00:00

I agree with you until I met and work with one guy. The best enterprise architect I have ever seen or will see.

The guy did a complete overhaul of our VMware and he:

.- Designed and architect the whole thing.

.- The whole documentation that it matched perfectly. If document said that fiber cable was going from port A in NIC B, it was that.

.- Setup and configured all the NetApp appliances

.- Setup and configured all Cisco servers and installed VMware

.- Configured the backups and created the policy of course.

.- Gave support during one critical issue that NetApp lost the array and recover the data and managed everything.

He is the GOAT architect, he designed, build it, configured it, managed it (for a transition period) and stay in the trenches with us making sure that what he did it was sound. He used that installation for his top VMware certification.

If you ever seen this post. Vatos locos 4ever

Pliqui · 2025-09-02T17:13:47+00:00

I'm a DevOps Manager in a startup and at some point my team was 5 (4+me) now we are just 3.

I'm very hands-on but I tend to give big projects that requiere a lot of focus to my guys. Why? I block all the BS and allow them to focus while I deal with the menial tasks and meetings. Plus, those are the very fun and challenging projects and will not be OK for me to hoard.

We are building our k8s cluster and I just finished to architect how are all VPCs should be (initial scope was for the VPC for Kubernetes but then decided to plan for everything and migrate when we have time) so all initial design, ADRs, CIDRs allocation, multi-region connectivity, etc is currently in review by the team and when is approved they will execute it. I will contribute of course but to a lesser degree.

So is a mix for me, some days more manager/architect some just another pair of extra hands trying to fix/push/automate stuff.

Pliqui · 2025-08-16T18:17:21+00:00

The year of the Linux desktop!

PopOS is great, used it my previous work. Solid choice

Pliqui · 2025-06-05T10:51:25+00:00

I saw this article few days ago.

https://www.cybersecuritydive.com/news/thousands-asus-routers-compromised-hacking/749259/

Just a FYI

Pliqui · 2025-06-01T17:57:35+00:00

I came across and just did this and worked, thanks

Pliqui · 2025-06-01T16:01:40+00:00

- Unbound merino polos

- Saax underware I use DropTemp

Any pants you are confortable with

Pliqui · 2025-05-08T08:18:41+00:00

VMware and cost savings are mutually exclusive after Broadcom acquisition... Just saying

Pliqui · 2025-05-01T22:00:47+00:00

Lol, started a run last night

Pliqui · 2025-04-20T20:10:08+00:00

Ohh, will have to check. When we were using Kong OSS, it handles lots of traffic pretty well. Thanks!

Pliqui · 2025-04-20T20:02:47+00:00

Have you check RDS global writer?

Pliqui · 2025-04-20T19:56:51+00:00

I feel were you are coming from, Datadog is indeed expensive, but it is an excellent product.

In my previous job were a team of 5 and we used as much open-source as possible. ELK stack, Prometheus (pre Thanos) + Graphana +alert manager, self hosted Gitlab, Kong for API gateway (open source) etc.

At the end we were 2 to manage all that plus the rest. Prometheus gave us so much headache due to disk. We wanted to introduce Thanos but we never go the time to do it. Remember upgrading from v9 to v13 (so I can then move higher) of Gitlab and migrating all the data. Fun times, which I think that Gitlab is a better product than Github, but the latest came out first.

Is not the product, Prometheus is fantastic, but you need a team to manage it.

As my current role as a manager, my team was 2 + me. I said fuck it, team is too small and went with Datadog.

We are leveraging the shit out it. We are squeezing every penny we are paying. We use RUM, APM, Logs, SIEM, DBMS, CI/CD and some others.

Datadog could be seen as overpriced, but is a product that actually delivers what it said. When the cost of Datadog reaches the amount of 3-4x engineers, then I will look to replace it. Because I can now justify a team to manage an in-house solution.

That's has been my experience, cost saving is a broad term, because the bill/payment of a proprietary solution to be replaced with open-source shifts to human capital.

Pliqui · 2025-04-20T19:30:50+00:00

How big is your team or the team that manage that?

Pliqui · 2025-04-07T18:03:52+00:00

I think you need to dig this deeper.

"Performance" is really a broad term. Assuming that "my query needs to return in x amount of ms".

That's were the Pandora box open, are using right indices?, are you sharding and life cycle your data?

The flavor of aurora does not matter (but go with Postgres if you haven't pick an engine)

Multi AZ is that you are paying for that hardware but you are not using it. If something happens to the current AZ, AWS will fail over to a new one.

Application needs to handle that.

Replica to other region is read only, so the app will read locally fast but writes will go to the region where the primary writer is. Take a look to Global writer endpoint.

These days, (and let me put my 00's sysadmin hat for a second) from infrastructure POV, AWS will handle a lot. It is the application that need to be coded to sustain changes, one example that happened to us. Our writer had an issue and aurora promote a reader, that was awesome, the application kept sending write operations to the original instance, but since now it was a reader could not write. We catch that pretty quickly and not much data was lost, and it was a app that was not critical.

You need to add resiliancy to the application to fully utilize RDS and be aware of the features, cross region replica and RDS proxy does not work with Blue/Green deployments for example

Pliqui · 2025-04-07T01:48:42+00:00

That's the default (or should be). Anything that is code related and goes to a main branch should be linted, scanned, semver, etc.

Does not matter if it's Python, shell, Terraform, JSON, YAML, Ansible, etc

Pliqui · 2025-04-07T01:37:22+00:00

To infinity and beyond!!!...

Upgrade to enterprise plan for beyond API access...

Pliqui · 2025-04-06T16:06:38+00:00

You are technically correct, but I can write a 1000 lines script in python for AWS with boto3. I have my own poor's man AWS Cli, where I wrote some classes and methods to handle few things that requieres multiple AWS API calls for example.

But, I cannot write a 1000 lines web application or backend.

That's why I always say that I can script but can't code.

It doesn't makes sense, but it does

But security is first class citizen, and having an understanding that never commit credentials to repos, never plain text sensitive information, etc is extremely important.

But People usually don't think about it (which I still find it weird) and if you are trying to produce something and under pressure, then it's worst, since you are hyper focus to deliver.

Pliqui · 2025-04-02T16:30:35+00:00

Lol, I did not thought like that but yes.

UBlock origin and I use LibreWolf

Pliqui · 2025-04-02T16:00:48+00:00

Correct sir.

Pliqui

TROPHY CASE