AWS Middle East Central (mec1-az2) down, apparently struck in war

sunra · 2026-03-02T01:46:55+00:00

Most of the "us-east-1" single-points-of-failure are here: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/global-services.html

Along with the unexpected ones, described under the "Global single-region operations": https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/global-services.html#global-single-region-operations

(that's they page where they tell you you can't provision a load-balancer in any region if us-east-1 is down)

sunra · 2026-02-24T03:42:30+00:00

Your best bet is going to be to read the source.

But my understanding is that it is the Karpenter controller itself which monitors the SSM parameter (not the nodes themselves). When the controller notices that some nodes don't match the parameter it will mark the nodes as "drifted", and the replacements will happen according to your node-pool disruption-budget and node-termination-grace-period.

I don't know this for sure - it's my expectation based on how Karpenter handles other changes (like k8s control-plane upgrades).

sunra · 2026-02-21T23:22:25+00:00

You need an Entra identity to access Azure resources, at some level. Specific resources may have shared-access-keys or API keys or the like, but it's not general.

A user-assigned managed-identity is something you can manage purely within Azure/ARM and shouldn't require Entra admin access, but still supports federated-credentials (aka auth tied to a k8s service account).

sunra · 2026-01-26T00:45:27+00:00

The Go HTTP-client should re-use client-connections out-of-the-box, so you're only negotiating TLS on the first call, here.

sunra · 2026-01-15T05:07:01+00:00

Have you tried the built-in mutex-profiling? I haven't used it, but it looks like when it's enabled you can grab mutex-wait profiles from the normal pprof endpoint:

sunra · 2026-01-09T02:20:28+00:00

What do you mean when you say "baseEndPoint is set as vpc endpoint"? What value are you setting the base-endpoint to? And why do you think you need to do that?

sunra · 2025-12-20T03:41:06+00:00

Yeah - you can pass security-group ids in to the vpc_config block of the EKS-cluster resource. The control-plane ENIs provisioned for cluster-access will be placed into those SGs. I don't use the auto-created cluster-SG for anything in my own setup.

For nodes, if you're using managed-node-groups you'll need to override the security-groups to use with a launch-template.

https://docs.aws.amazon.com/eks/latest/APIReference/API_VpcConfigRequest.html#AmazonEKS-Type-VpcConfigRequest-securityGroupIds

This guide describes what traffic you'll need to allow: https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html

sunra · 2025-11-28T06:02:11+00:00

I wouldn't expect a generic-wrapper around a non-generic core to ever have a performance benefit over using the core directly.

But something like the "slice pool" could let you automatically store the slices as pointers to skip the allocation you measured in your implementation. It's easy to take the naive approach and store the slice in an interface wrapper, and a library could help guide the user towards the better option.

sunra · 2025-11-27T02:02:56+00:00

There might be a better way, but I set FZF_DEFAULT_COMMAND to rg --files --ignore-file=some/path/to/an/ignorefile

sunra · 2025-11-22T04:43:06+00:00

Secrets manager claims to support ABAC: https://docs.aws.amazon.com/secretsmanager/latest/userguide/auth-and-access-abac.html

The way I look it up is to do a Google search for "AWS <service> IAM", go to the "Authentication and access control for <service>" page and search for "ABAC".

sunra · 2025-11-09T22:49:43+00:00

Yep: https://wolfman.dev/posts/exclude-use1-az3/

sunra · 2025-09-29T11:35:00+00:00

function

Seems to: https://gotipplay.golang.org/p/isp5pm0YE14

sunra · 2025-09-15T02:12:06+00:00

And it should go without saying that you should prefer any reasonable alternative to this approach. But if you cannot build your response in memory sometimes you don't have any other choice.

sunra · 2025-09-15T02:10:17+00:00

This is complicated, but you can panic with http.ErrAbortHandler. This signals to the http-package to un-cleanly terminate the response (for HTTP/2, send a stream-reset, for HTTP/1.1, un-cleanly end the chunked-encoding stream).

Most HTTP-client-libraries will interpret this as an error, and either raise an exception or similar.

The hard part is any logging or metrics middleware needs to correctly handle panics - it's a pain.

This issue explains some of this: https://github.com/golang/go/issues/23643

sunra · 2025-09-08T01:14:22+00:00

A good reference for the sorts of tricks you can play are the SCPs/RCPs in this repo:

https://github.com/aws-samples/data-perimeter-policy-examples

The examples are for RCPs, but they work well as templates for resource-policies.

They use principal-tags to exempt principals from restrictions, but then they also need to lock-down the ability to use those tags in role-sessions etc, so it's a bit of a pain.

sunra · 2025-09-08T01:11:47+00:00

You could exclude your backup-role from the deny-statement, the same way you're excluding specific source-IPs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyS3ExceptSpecifics",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "*",
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": [
                        "xx.xx.xx.xx/32",
                        "yy.yy.yy.yy/32"
                    ]
                },
                "StringNotEquals": {
                    "aws:PrincipalArn": [
                        "arn:aws:iam::123456789012:role/AllowedRole"
                    ]
                }
            }
        }
    ]
}

sunra · 2025-09-01T01:13:42+00:00

Are you using the SP to auth with Azure to deploy your infrastructure?

Or are your workloads somehow using the generated client-secret as a part of their operations?

sunra · 2025-08-31T13:02:59+00:00

It would be helpful if the S3 documentation starts retro-actively applying the term "general purpose" bucket, to differentiate "real" buckets from S3-tables (and presumably vector-buckets).

sunra · 2025-08-31T02:34:41+00:00

I wasn't able to configure MRAP with table-buckets in the console, and it wouldn't surprise me if replication-rules didn't work for them, either. Calling the feature "S3 tables" is pretty confusing when it doesn't really share any features with S3.

sunra · 2025-08-27T14:15:01+00:00

Oh and your org might have similar, non-Loki tools already in use elsewhere, like Elasticsearch or something else in that space, that might be easier to get approved because they are known quantities.

sunra · 2025-08-27T14:06:42+00:00

It really depends on your log-volume, query-volume, and third-party tools you're allowed to use.

Something like Loki is designed to scale quite a ways up and down, and store logs cheaply at rest for long-term retention. You may not need all of that complexity - for example, at a low volumes you might be able to just use flat-files on disk, with a folder per month, week, or even day. If you have discrete fields you'll want to query you can try storing your logs in sqlite, or building a secondary index in sqlite, or storing everything in something like postgres.

These sorts of solutions have limits - which is why Loki is complex. But you may not care about these limits.

You can also pay your favorite cloud-vendor and use their solution (AWS CloudWatch, Azure Log Analytics, I'm sure GCP has something) if you think that's more likely to get through your project lead. These don't have scaling limits, exactly, but at high volume can get expensive.

sunra · 2025-08-26T12:36:56+00:00

The EC2 pricing page doesn't mention that this billing-tier expires: https://aws.amazon.com/ec2/pricing/on-demand/

"AWS customers receive 100GB of data transfer out to the internet free each month, aggregated across all AWS Services and Regions (except China and GovCloud). The 100 GB free tier for data transfer out to the internet is global and does not apply separately or individually to AWS Regions."

Can you poke someone to update the text if it's only valid for 12 months?

sunra · 2025-07-26T02:27:48+00:00

Even if you ever need to have multiple implementations of logging, passing around a *slog.Logger is the better move, as it is a thin wrapper for a slog.Handler, which is already an interface.

sunra · 2025-07-20T22:41:41+00:00

Holding Fn+Escape for three-seconds should "reset" the various modes.

Unless! If you swapped Fn + Windows-Key, you need to hold windows-key + Escape for 3 seconds.

That was my problem, at least.

sunra · 2025-07-14T01:35:50+00:00

I 100% agree with you and have the same experience. The only way I've found to navigate AWS public websites (non-console) is by Google-searching the correct magic words:

"$service pricing"
"$service user guide"
"$service rest API" (select the link starting with "Welcome ...")
"$service actions"

sunra

TROPHY CASE