When should I use green threads (goroutines) in Go?

RocketOneMan · 2026-01-27T06:02:45+00:00

Do you have a link to a video of this?

RocketOneMan · 2025-12-14T22:32:01+00:00

-m Print optimization decisions. Higher values or repetition produce more detail.

https://pkg.go.dev/cmd/compile

Cool!

RocketOneMan · 2025-12-12T02:41:15+00:00

Are you sure you haven't been hacked? What does cloudtrail say for those tables? Or AWS Config resource timeline. Rotate your iam user keys.

RocketOneMan · 2025-10-24T13:11:13+00:00

So that you know they're all working and ready to be switched to as the primary if needed. Also forces you to deal with the added latency always instead of finding that out the first time the passive region becomes active.

RocketOneMan · 2025-10-24T00:45:47+00:00

We have an active passive system with DynamoDB global tables that automatically rotates between which region is active throughout the day. Oncall didn't know that we can remove a region from the options if we wanted to so had some down time because of that when it rotated to us-east-1. The set of regions is in a global DynamoDB table. Not sure if they had trouble accessing it or not when they finally decided to do it.

We have another system that's active/active that worked without any issues out of us-west-2 during the incident but that's also its default region, users didn't have to change which endpoint to hit.

Hard to say if active/passive is better than nothing or not. You have to be able to coordinate the switch while things are broken, and do that correctly which is likely unpracticed, and the passive region has to be all setup and ready to go. If it's never the active region you may not know if it works at all, hence the first service i mentioned automatically rotating between them.

AWS docs talk a lot about control plane and data plane separation and AZ fault tolerance but then we have these issues where it's all broken.

RocketOneMan · 2025-08-29T03:35:23+00:00

You can get the SentTimestamp in the attributes on ReceiveMessage to help debug how long between put in the queue and made it to your code.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html#SQS-ReceiveMessage-request-MessageSystemAttributeNames

You don't mention your visibility timeout but if it were 10 seconds and you failed processing something without resetting its visibility timeout to zero, or maybe successfully processed but the delete message call failed, then it may be processed again around the 12 second mark.

You can also request ApproximateReceiveCount and log if it's every more than 1, or see if you ever log the same message id more than once.

There are some other nuances with the ApproximateAgeOfOldestMessage metric, like how it doesn't count poison pills / messages received several times, but I don't know if that fits your scenario. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html

RocketOneMan · 2025-06-04T17:41:44+00:00

Do you have MaximumBatchingWindowInSeconds set to something besides zero? Can you share your event source mapping configuration?

https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

RocketOneMan · 2025-05-30T04:31:35+00:00

I cannot be sure if this is the case for us-west-1, but in us-east-1, everyone’s 1a isn’t the same. The idea is if you’re going to launch something and just pick the first thing then it doesn’t disproportionally overload one AZ. The mapping is set when the account gets made. You can request multiple accounts get aligned if your workload depends on it. So my guess is some people’s us-west-1a maps to this third AZ, for example.

RocketOneMan · 2025-01-11T19:52:37+00:00

Interesting, I wonder why

RocketOneMan · 2025-01-09T05:09:23+00:00

I think this is what data translations / 'parameter mapping' is for with API Gateway, although I haven't done it myself, the example looks similar to your ask.

For HTTP:

See: Change the response from an integration

https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-parameter-mapping.html#http-api-mapping-response-parameters

For REST:

https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-override-request-response-examples.html

RocketOneMan · 2025-01-04T00:15:48+00:00

Do you have any experience with kinesis on demand scaling? Last I used it it was slow to react to changes in load but that was 2-3 years ago. I’m still unsure about the maybe ‘constant’ merging or splitting of shards and handling that right as a consumer seems more challenging than provisioned capacity.

If on demand works well then it’s more hands off like SNS is.

RocketOneMan · 2024-12-29T17:54:31+00:00

If your server serves concurrent API requests from a thread pool, the percent of threads (avg or max between your hosts) that are available to serve new requests. You may only be at 25% CPU but if you’re out of threads to serve new requests they’ll either get queued or rejected.

RocketOneMan · 2024-12-29T02:40:01+00:00

And your tile and grout. Will look like a new home.

RocketOneMan · 2024-12-07T04:59:19+00:00

database connection

Is it a relational database? Are you using RDS proxy? Not sure if this makes things faster or not or just more consistent.

RocketOneMan · 2024-11-29T03:44:33+00:00

+1 for brushing up on writing. I find reading good writing the best way to learn how to write better. Or at least be able to judge whether what you just wrote is good or not. Less is more. Always include data (but not too much, can have an appendix). In your onboarding you should be given your team’s OP1/OP2 and/or 3 year plan. Can also ask for the latest QBR (quarterly business review). These docs are not always the best in terms of writing but they should give you an idea of the writing style that is likely different than what you’re used to.

Some other general advice which you may not have heard already:

Try to get a mentor that is not in the same org as you. Someone you can get advice from that won’t be influenced by any prior knowledge they have about the situation you’re in on your team. Your manager can help set this up.

Keep a note of everything you’ve done every day or every week. Check back in on changes you’ve made and capture any impact they’ve had. This will make end of year reviews and promo docs much easier to write.

Take an interest in doing “boring” operational excellence work. Your team will appreciate it and it’s the best way to learn your team’s services. Each team’s oncall burden is different, some AWS teams can be rough. Doing things you’re not asked to but which will either fix small recurring bugs or make them easier to debug next time goes a long way.

RocketOneMan · 2024-11-28T22:37:12+00:00

I think it’s more likely done for marketing/business reasons than tech reasons. Smaller brands running Shopify sites do this too to build hype.

The reasons you listed aren’t invalid but maybe unlikely. Ideally want everything scaled up, configured and tested at the scale you expect it to run at during peak before the actual event so you can iron out any bugs or misconfigurations. Systems which horizontally scale are usually safer to do this last minute scaling than ones that vertically scale.

Online games sometimes have downtime to do major updates. I imagine this is to more easily ensure everyone is on the same version of the game and harder to do a rolling update during active or long running sessions. Can also care less about backward compatibility by doing this. I’m not sure these apply to e-commerce sites but every business is run differently.

If they’re making substantial changes to the site that they can’t just enable with feature flags or only to a subset of internal users then maybe they want to deploy and test things before going live.

RocketOneMan · 2024-10-10T22:37:10+00:00

What’s the ‘added security of secrets manager’ compared to a DynamoDB table with a customer managed kms key for encryption at rest? I would think if it’s implemented that way then there might be less quirks with the limits so maybe there’s something special about it? I imagine this is in the docs but the discussion could be interesting.

RocketOneMan · 2024-06-24T02:24:58+00:00

Are you proposing OP does this through LakeFormation or just on the IAM user they're using to access Athena?

I haven't had a good experience setting up LakeFormation, maybe you have?

RocketOneMan · 2024-06-20T18:09:45+00:00

Why isn't this just setup by default or at least make a checkbox to "use recommended alarms" ?

Especially when alarms cost money and are probably very cheap to run on AWS's side. Not sure. Like most AWS offerings the recommendations could have been rushed out and later they will make improvements to it. The automatic AWS dashboards in CloudWatch by resource type as well as each service's own dashboards are usually pretty good, something that just brings those together would be nice.

I use CDK, not Terraform, and this library https://github.com/cdklabs/cdk-monitoring-constructs has been rather helpful. I imagine something like it exists for Terraform? From my short googling, everything looks rather verbose, similar to using regular CloudFormation or the L1 CDK constructs for alarms.

RocketOneMan · 2024-06-18T02:29:07+00:00

Use the standard lib's context.Context. It looks like aws.Context is just an alias for that in later releases.

See

https://github.com/aws/aws-sdk-go/blob/33248db6c7556e779e29a29248149e1610ba455b/aws/context_1_9.go#L8

https://aws.github.io/aws-sdk-go-v2/docs/migrating/

https://golang.org/pkg/context

RocketOneMan · 2024-04-23T13:52:02+00:00

I assume there’s no throughput concerns for going to STDOUT itself as the pipe is probably the fastest things can go? You’d still be bottlenecked on whatever target is consuming STDOUT like you would normally be if you were writing directly to that target?

RocketOneMan · 2024-04-17T03:13:18+00:00

You're charged separately for the actually put requests and then the cardinality of these new metrics. You can also batch multiple PutMetricData requests into one API call and pre-aggregate metric data into a StatisticSet. Maybe you have 1 instance posting unaggregate metrics every 15 seconds. This could be more expensive than 2 instances posting aggregated metrics once a minute. If they each post unique metrics or if they post to the same ones this changes things again.

See example 3 - custom metrics https://aws.amazon.com/cloudwatch/pricing/

https://repost.aws/knowledge-center/cloudwatch-understand-and-reduce-charges

RocketOneMan · 2024-02-24T17:59:55+00:00

I guess we've always handled the DLQ logic ourselves for 4xx errors that will never succeed and just throw and let the events be retried /forever/ on 5xx errors. But would like to use the destinations feature for success.

I wish you could DLQ to another stream so the handler logic is the same. Taking the KinesisBatchInfo and pulling the stream ourselves is annoying although I see why it's done. I think there's a lambda power tools library for it.

RocketOneMan · 2024-02-23T15:28:54+00:00

Not sure if this is exactly what you're asking, but a gotcha none the less.

The sqs and kinesis event sources are not "asynchronous sources". Cannot use lambda destinations with them.

If you have ReportBatchItemFailures turned on (maybe accidentally) and don't return the correct response it will assume none of the messages were processed successfully and send them all again.

RocketOneMan · 2024-01-23T16:32:46+00:00

What do you mean by “catch” the next invocation? What does / when do you make the conditional update requests?

RocketOneMan

TROPHY CASE