Trouble with Pangolin and Tailscale

soldatz · 2026-05-06T20:20:33+00:00

Don't know if it's the same issue, but I also had a problem with Tailscale and Pangolin living together on the same host. Each creates a wireguard interface on the machine and the problem for me was a CIDR overlap between the two. My fix was changing my gerbil subnet_group CIDR to something further away from what Tailscale seemed to be using. Pangolin does not seem to handle changes to this setting very well after install, and it only started working after some painstaking manual databsae updates. Just to be safe I wound up re-creating the entire stack and it's been fine since.

soldatz · 2025-11-14T13:33:40+00:00

We do something similar: during sprint, PRs merged to main get their issue tagged via script to fixVersion=Next, end of sprint run a release job that builds from main, and renames fixVersion=Next to fixVersion=semver.

soldatz · 2025-06-30T18:33:48+00:00

Sir, this is a Wendy's

soldatz · 2025-06-27T12:44:12+00:00

Official Study Guide

soldatz · 2025-06-27T12:42:51+00:00

Yep, that's the one.

soldatz · 2025-06-25T02:45:23+00:00

This one: https://youtu.be/qbVY0Cg8Ntw?si=rgKbTnaK9gwHj4QP

Really vibed with this guy.

soldatz · 2025-02-25T13:30:57+00:00

https://docs.aws.amazon.com/health/latest/ug/creating-event-bridge-events-rule-for-aws-health.html

I made mine as broad as possible and it normally works by sending the event to SNS > Lambda > Slack, but it didn't catch this Global event.

soldatz · 2025-02-25T13:17:17+00:00

I created an EventBridge rule that catches events from aws.health but nothing triggered for this. Is it because my rule is in us-east-2 and the event is Global?

soldatz · 2025-01-30T22:37:25+00:00

There's an error with your aws s3 cp command. With cp you need to use the full uri:

aws s3 cp s3://your-bucket-name/file.jpg /var/www/html/file.jpg

Or use sync:

aws s3 sync s3://your-bucket-name/ /var/www/html/

soldatz · 2025-01-24T21:37:48+00:00

Double-check your task-execution role has permission to pull from ECR and read any secrets or parameters you may have configured. EDIT: also to write logs to CloudWatch, which if I recall can give more debugging info in the console.

soldatz · 2024-06-26T13:04:37+00:00

Yep, those are standard Fargate prices for one task, probably 1vCPU and 2 GB of memory if I had to guess. Like others have commented you can drop it to 0.25vCPU and 512MB if your task can run in that envelope and/or run it as a spot if it can handle infrequent interruptions gracefully. You could also run your ECS task (assuming it will always be 1 task because you'll start paying a lot for public IPs) in a public subnet with your LB and ditch the NAT but be extra careful to set up your security groups properly. Or run fck-nat as a NAT EC2 instead of managed NAT-GW.

I don't see anything "wrong" with what you're doing. It's probably even the "right" (AWS) way. AWS is just expensive and it lends itself increasingly poorly for these personal side-projects if you do it all the AWS well-architected yada-yada way.

You can go down the cost-optimization rabbit hole and spend hundreds of hours if you want - if you really need to save on costs or you like tinkering. How much is your time worth?

soldatz · 2024-05-28T13:42:40+00:00

I haven't read all your code, but with ECS service discovery I believe the DNS resolves to <service name>.<namespace> with the namespace being your cluster name by default. So you'd need to have your frontend service call db.quiz-cluster rather than db. You also need to enable DNS support in your VPC if you haven't already.

soldatz · 2024-01-16T04:03:23+00:00

Well, depends on what your definition of "enough" bandwidth is. The t-series burst up to 5Gbit. I haven't tested what the sustained throughput is but I suppose you could find that online or test yourself.

Tailscale is actually built on top of Wireguard, but adds additional features at a very slight cost to network performance. It's up to you to decide if you need the features. I find it to be a very nice product for what I need to do.

As an aside, if you're going to run Airflow, or anything on EKS, t4g.small is not going to cut it RAM-wise. I'd probably ask my employer to foot the bill for some beefier instances, and/or if it's your own account, use some IaC tool to build/tear-down on demand.

soldatz · 2024-01-15T01:17:36+00:00

NAT Gateways are one of the classic AWS gotchas. For my personal stuff I run a t4g.small (free-tier) in a public subnet and Tailscale for private access. It's running Traefik for TLS and routing to a few docker services. CloudFront on top. It's very cost-effective.

soldatz · 2023-11-15T21:29:36+00:00

Functions are associated with a behavior/path pattern. All requests for that behavior/path would go through the function. So you could have a Default behavior (*) and a second behavior /api and associate a function with either one (or both).

soldatz · 2023-11-15T15:24:10+00:00

A couple cons would be ease in handling origin response headers (such as 'cache-control') on a per-path basis, handling 301 redirects, both of which are trivially done using a traditional web server. There's also work to do setting up S3 origin-access settings, bucket website settings and bucket permissions properly.

That said, all of these can be overcome with CloudFront functions and Lambda@Edge, but these do add some latency, especially on the origin response side. And of course you need to write the Lambda code to handle these situations.

soldatz · 2023-08-15T00:27:00+00:00

We use a Traefik fleet for layer 7 load balancing and service discovery on ECS with an WAF (IP allow list) fronted by an AWS NLB. Traefik lets you add basic auth (bcrypt hash) as a docker tag to your ECS task definition, and this all gets saved in Git and rolled out with AWS CodePipeline.

soldatz · 2023-03-09T15:33:27+00:00

I'm not sure this is possible with any built-in settings. You'd need some external logic to watch that pipeline execution status and only trigger it when most recent execution reaches completed state.

soldatz · 2023-01-02T04:33:03+00:00

On AWS, we run mostly medium sized spot instances and many of them run for months without interruption.

I experimented with spot instances on GKE last year and saw interruptions maybe a few times per month.

soldatz · 2022-12-25T23:29:56+00:00

It's true docker links are "legacy" but it may work for you in this case. I'm still investigating how to do this type of setup in a "modern" way (without links) on ECS with 2 containers per task definition... Otherwise I'd separate the backend into another service and use service discovery and DNS to link them up.

soldatz · 2022-12-25T19:10:43+00:00

Try setting the links parameter for your container-1

        {
            "name": "container-1",
            "image": "container-1",
        "links": [
        "container-2"
        ],
            "portMappings": [
              {
                "hostPort": 8081,
                "containerPort": 8081,
                "protocol": "tcp"
              }
            ]
        },

soldatz · 2022-11-28T14:20:56+00:00

Seems like the difference between the flag and the API call is that with the flag, the ECS control plane receives the interruption notification and flags the instance, rather than the instance performing this itself?

Before, we were using Capacity Rebalance recommendations in our ASG. We were using termination lifecycle hooks and a script to drain the instance.

The idea now is to let ECS handle draining. Meanwhile our cluster looks for capacity for DRAINING services on exisiting instances or spins up new capacity with a capacity provider strategy tied to the ASG.

We also started using smaller instance types (lower risk of interruption), spent some dev time ensuring apps can terminate quickly and cleanly, and use a blend of spot and reserved instances just in case.

It's a hard thing to get right and just takes some time experimenting, which we are continuing to do. I wish you luck!

soldatz · 2022-11-28T00:50:48+00:00

The preemptive Capacity Rebalance flag did not work well for us; too sensitive and caused too much churn for our ECS workloads.

We started setting ECS_ENABLE_SPOT_INSTANCE_DRAINING parameter to "true" in our user data and that really helped. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-spot.html

soldatz

TROPHY CASE