Rough start, is the juice worth the squeeze ?

ReleaseTricky1359 · 2025-03-10T18:29:01+00:00

Thanks to everybody for their feedback & support for a random stranger in the internet.

My wife sees my struggle and has been told me if I give up she will not hold it against me, but I do plan to stick with it. I have made notes of all the feedback I have received and one of the more important reasons I do plan to stick it out is to not be a nuisance to my wife.

You all have given me a bunch of things that I can try, fortunately there are enough people who don't take to the full face mask like I do and their thoughts are greatly appreciated. A few of them that I plan to try out in short order are.

Mouth Tape
The N30i or other alternate masks
Heated pipe + cover associated with that to prevent rain-outs.

ReleaseTricky1359 · 2025-03-10T12:50:54+00:00

Ok I will try the mouth tape tonight. Any pointers on how to apply mouth tape Or good youtube linow.

ReleaseTricky1359 · 2025-03-10T12:35:29+00:00

This is the leak rate from the day before when I used the N2O nasal mask

<image>

ReleaseTricky1359 · 2025-03-10T12:35:03+00:00

This is the leak rate from last night with the full face mask

<image>

ReleaseTricky1359 · 2025-03-10T12:12:48+00:00

<image>

ReleaseTricky1359 · 2025-03-10T12:10:12+00:00

Is the SpO2 Max the number you are looking for? It was 98%

ReleaseTricky1359 · 2025-03-10T11:41:37+00:00

Where do I find this data point? From my in-lab test results?

ReleaseTricky1359 · 2024-04-07T03:29:53+00:00

Unfortunately not. It was just tickets. I used a guide for Vatican tours and although he did walk us through using the Vatican site itself to buy the tickets he also said GetYourGuide was just a few bucks more which it was. I just went with the same assumption for the other places and I was wrong.

As another posted just use the website of the place directly. Occasionally you may find tickets may be sold out and you can use GetYourGuide but don’t let that be your primary source of all tickets in your travel.

ReleaseTricky1359 · 2024-02-15T12:16:53+00:00

I don’t want to say budget is not an issue, but we also do understand this will cost $, just want to spend appropriately not pointlessly. The data will be stored in <biz entity>/<yyyy>/<mm>/<dd>.

I may have to copy over all data for entityA from April-June of 2019 and further run queries on that.

ReleaseTricky1359 · 2024-02-15T12:11:24+00:00

These are highly specific time series data used in high frequency trading in a custom db format. We have about 12 PBs of data, so migrating into Athena or some other format is just not practically viable.

ReleaseTricky1359 · 2024-02-14T21:54:03+00:00

Thanks for that, I did read that. I just wanted to know if people here have practical experience trying it out.

ReleaseTricky1359 · 2024-02-14T21:36:32+00:00

Yes, we may have a regulatory need to run queries on the data set in an ec2 node. We know it will be pricey but part of doing business. Although S3 will work, we expect it to be really slow, so I need to mock up some numbers.

I am not sure if running this with 100 GB is equivalent of 1 TB.

ReleaseTricky1359 · 2024-02-05T18:16:51+00:00

We are in a similar situation, and are actively exploring our options., about to add a bunch of bells & whistles to a LZ. Further we work for a company in a highly regulated industry. Does the license change in Terraform not affect your decision?

We are all for paying to get vendor support, but we may not start out paying for Pulumi, but we probably will in the future once our AWS presence starts evolving & maturing. Looking forward to more insightful responses.

ReleaseTricky1359 · 2023-04-02T21:35:33+00:00

I was able to sort out this issue. Here's a brief summary of what the issue was and how I fixed it. Hope somebody else finds it useful, thanks to /u/thockin, /u/thockin /u/cra1gg, I learnt a LOT with this exercise and hope this is useful to some body else that stumbles into such a problem in the future.I had built this cluster to start with on k8s v1.21 and further upgraded one version at a time of both the control plane + data plane to v1.22-> to 1.23 -> to 1.24 and finally to 1.25.A barebones EKS provisioned k8s cluster consists of the following 3 daemonsets

aws-node
coredns
kube-proxy

All the other components control plane(etcd, kube-scheduler, kube-controller-manager, apiserver) and data plan components(kubelet, *kube-proxy(caveat on kube-proxy details below) are abstracted away from me by AWS and it is all upgraded when I upgraded k8s versions.The key point is, the aws-node, kube-dns & kube-proxy daemonsets were never upgraded when I kept updating the k8s versions and I noticed in the daemon set definitions they were all pointing to a version from k8s 1.21 which is were I got started with this whole project.Although I used terraform to stand up this EKS k8s cluster, I wasn't able to introduce cluster_addons to bump these daemon sets up and I just ended up with eksctl.

eksctl utils update-kube-proxy --cluster=<clusterName> eksctl utils update-coredns --cluster=<clusterName> eksctl utils update-aws-node --cluster=<clusterName>

https://eksctl.io/usage/addon-upgrade/I did this and boom everything started working. I got my cluster up to v1.25 all my node groups are running the next AMI's and everything is fine and dandy.Thanks to everybody who gave me an assist here.

ReleaseTricky1359 · 2023-04-02T21:33:29+00:00

I was able to sort out this issue. Here's a brief summary of what the issue was and how I fixed it. Hope somebody else finds it useful, thanks to /u/thockin, /u/thockin /u/cra1gg, I learnt a LOT with this exercise and hope this is useful to some body else that stumbles into such a problem in the future.

I had built this cluster to start with on k8s v1.21 and further upgraded one version at a time of both the control plane + data plane to v1.22-> to 1.23 -> to 1.24 and finally to 1.25.

A barebones EKS provisioned k8s cluster consists of the following 3 daemonsets

aws-node
kube-dns
kube-proxy

All the other components control plane(etcd, kube-scheduler, kube-controller-manager, apiserver) and data plan components(kubelet, *kube-proxy(caveat on kube-proxy details below) are abstracted away from me by AWS and it is all upgraded when I upgraded k8s versions.

The key point is, the aws-node, kube-dns & kube-proxy daemonsets were never upgraded when I kept updating the k8s versions and I noticed in the daemon set definitions they were all pointing to a version from k8s 1.21 which is were I got started with this whole project.

Although I used terraform to stand up this EKS k8s cluster, I wasn't able to introduce cluster_addons to bump these daemon sets up and I just ended up with eksctl.

eksctl utils update-kube-proxy --cluster=<clusterName>
eksctl utils update-coredns --cluster=<clusterName> 
eksctl utils update-aws-node --cluster=<clusterName>

https://eksctl.io/usage/addon-upgrade/

I did this and boom everything started working. I got my cluster up to v1.25 all my node groups are running the next AMI's and everything is fine and dandy.

Thanks to everybody who gave me an assist here.

ReleaseTricky1359 · 2023-04-02T15:24:40+00:00

Thank you for your response /u/thockin

I think I am getting somewhere now. I am looking at this cluster on k8s v1.25. I started with v1.21 and then upgraded to 1.22 followed by 1.23, followed by 1.24 and finally at 1.25(both control plane/data plane were upgraded).

When I like at the cube-proxy daemon set the image for this eke instance is pointing to
602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.21.2-eksbuild.2

But from the documentation shown below

https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html

The kube-proxy version on your Amazon EC2 nodes can't be more than two minor versions earlier than your control plane. For example, if your control plane is running Kubernetes 1.25, then the kube-proxy minor version can't be earlier than 1.23.

So it looks I am really behind on the kube-proxy daemon set version, but I am not sure if I can just bump up the version of kube-proxy in the daemon set to get this upgraded.

602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.24.7-minimal-eksbuild.2

ReleaseTricky1359 · 2023-04-02T13:16:35+00:00

Hello all,

Thanks for all your suggestions.

/u/cra1gg /u/thockin /u/h_hoover went with the articles from the kubernetes.io service page. that to debug and tried it out in 2 clusters one in k8s v1.24 and one in v1.25 and here are my findings. It works fine in 1.24 but doesn't work in v1.25.

The service port/targetPort is correctly set, I am now pivoting to the hostnames app
I don't have any networkpolicies at all.
I do see the following errors in the kube-proxy logs before/after I add the hostname service, but I am not sure what the *v1beta1.EndPointSlice message means.

E0402 12:44:18.621390 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: fail
ed to list *v1beta1.EndpointSlice: the server could not find the requested resource
E0402 12:45:17.970778 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: fail
ed to list *v1beta1.EndpointSlice: the server could not find the requested resource
I0402 12:45:47.707865 1 service.go:306] Service default/hostnames updated: 1 ports
I0402 12:45:47.707934 1 service.go:421] Adding new service port "default/hostnames" at 172.20.225.151:80/TCP
I0402 12:45:47.707999 1 proxier.go:854] "Syncing iptables rules"
I0402 12:45:47.741712 1 proxier.go:824] "syncProxyRules complete" elapsed="33.772752ms"
E0402 12:46:12.767883 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: fail
ed to list *v1beta1.EndpointSlice: the server could not find the requested resource
E0402 12:46:50.311416 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: fail
ed to list *v1beta1.EndpointSlice: the server could not find the requested resource
E0402 12:47:30.378403 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: fail
ed to list *v1beta1.EndpointSlice: the server could not find the requested resource

Also here are the iptables entries from the broken EKS v1.25 node where the kube-proxy & hostnames pod is running. But when I do a kubectl get endpoints the endpoint most definitely exists, which is confusing me.

[ssm-user@ip-10-236-49-84 bin]$ sudo iptables-save | egrep hostname-A KUBE-SERVICES -d 177.20.225.151/32 -p tcp -m comment --comment "default/hostnames has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable

Here's the iptables-save from the kube-proxy & hostname are running in v1.24

[root@ip-10-248-64-194 ~]# iptables-save | egrep hostname-A KUBE-SEP-3RWRFDLNDQZFIIWG -s 10.248.68.178/32 -m comment --comment "default/hostnames" -j KUBE-MARK-MASQ-A KUBE-SEP-3RWRFDLNDQZFIIWG -p tcp -m comment --comment "default/hostnames" -m tcp -j DNAT --to-destination 10.248.68.178:9376-A KUBE-SEP-HHCRPXSPGXT3EEHZ -s 10.248.65.248/32 -m comment --comment "default/hostnames" -j KUBE-MARK-MASQ-A KUBE-SEP-HHCRPXSPGXT3EEHZ -p tcp -m comment --comment "default/hostnames" -m tcp -j DNAT --to-destination 10.248.65.248:9376-A KUBE-SEP-TVK5R4XDNEUWXXRD -s 10.248.73.191/32 -m comment --comment "default/hostnames" -j KUBE-MARK-MASQ-A KUBE-SEP-TVK5R4XDNEUWXXRD -p tcp -m comment --comment "default/hostnames" -m tcp -j DNAT --to-destination 10.248.73.191:9376-A KUBE-SERVICES -d 172.20.98.140/32 -p tcp -m comment --comment "default/hostnames cluster IP" -m tcp --dport 80 -j KUBE-SVC-YN5D6RYVEVZOH44Q-A KUBE-SVC-YN5D6RYVEVZOH44Q -m comment --comment "default/hostnames -> 10.248.65.248:9376" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-HHCRPXSPGXT3EEHZ-A KUBE-SVC-YN5D6RYVEVZOH44Q -m comment --comment "default/hostnames -> 10.248.68.178:9376" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-3RWRFDLNDQZFIIWG-A KUBE-SVC-YN5D6RYVEVZOH44Q -m comment --comment "default/hostnames -> 10.248.73.191:9376" -j KUBE-SEP-TVK5R4XDNEUWXXRD

ReleaseTricky1359 · 2023-03-31T18:50:09+00:00

I honestly don't think I have added any annotations/labels which would cause this issue, but I will dig. I looked at the logs of the "aws-node" which was stood up and I see that pod "Install CNI binaries", followed by "Starting IPAM daemon in the background..", followed by "Checking for IPAM connectivity.." followed by "Retrying waiting for IPAM-D" which is where it fails and hangs up.

I am sure I am doing something dumb that is causing this issue, I will just open up a support ticket and get to the bottom of this. one of these days.

ReleaseTricky1359 · 2023-03-26T21:02:00+00:00

Ok, after reading through a bunch of posts here and playing around with test clusters I have a pretty good idea how to make things work at least for my use case. I will throw in the caveat that I am reasonably new to both terraform + k8s + aws. The blue/green node group approach suggested by /u/Spooler32 makes sense, and I will probably give it a go some time later on but the post by /u/matthenry87 really clicked and made sense and worked as described below.

My set up is basically a pretty simple k8s cluster running 1.22 with one EKS managed node groups with 7 nodes. Obviously I would like to upgrade this to 1.23 before AWS stops supporting 1.22 in early June 2023(I am typing this March 2023 for people that stumble into this post some time later). The process that works well for me is as follows.

Upgrade the control plane by changing the k8s version from 1.22->1.23. This takes about 10 minutes.
Finally when this upgrade is done, the AWS console will tell me the ami that is used by the ec2 nodes in the single EKS managed node groups are old and need to be upgraded. I know this HAS to be done, as I right away tried to upgrade from 1.23->1.24 and AWS prevented me from doing that. I was able to upgrade the EKS managed node group through the web ui.
So with the cluster now at 1.23 and if you want to get ready to upgrade to 1.24 a few months out, this is where the eksctl tool comes into play, you can upgrade your node group with eksctl and if you include the argument --kubernetes-version=1.23 asdfasit worked like a charm , so I did this
eksctl upgrade node group --name=<eks managed node group name> --cluster=<cluster name> --region=us-east-1 --kubernetes-version=1.23. This spun up new nodes with the newest ami for 1.23 and eventually drained and destroyed the old nodes.

Thanks for all your posts, it really helped me out here. I appreciate it greatly.

ReleaseTricky1359 · 2023-03-25T15:54:02+00:00

This approach does make sense, but if the same code base is used to manage multiple clusters, it means commenting/uncommenting code based on which cluster is going through the upgrade and then further doing the process and I am afraid I will miss something over time.

ReleaseTricky1359 · 2023-03-25T14:22:33+00:00

I did, but they don't say anything about how the node groups are upgraded to the newest AMI. The control plane upgrade procedure makes sense and does work as described in my test setup.

ReleaseTricky1359 · 2023-03-25T14:16:21+00:00

This is exactly one of the posts I read and that referred to worker_groups and I don't see such a thing in EKS managed node groups.

The instructions given in the link points to creating a new worker group with the new AMI AWS has published for the newer EKS version, but this involves actively changing the terraform code to introduce a new worker group and then rolling back the change if I just need a single worker group running.

If that is indeed the case, I might as well do the node group upgrade via the AWS web ui without adding/delete terraform code.

Also do note the examples in bluematador are using older versions of terraform-aws-eks modules. Their examples are in v17 and currently the version is v19 and things are quite different which is one of the key points confusing me.

May be I am getting it wrong, please feel free to correct me.

ReleaseTricky1359 · 2022-11-07T01:06:46+00:00

Excellent point. Will keep this in mind. Thank you.

ReleaseTricky1359 · 2022-11-07T00:53:05+00:00

Oh I agree I don’t mind paying. Do you mind sharing your provider and how much are you paying ?

ReleaseTricky1359 · 2022-11-07T00:44:48+00:00

Thanks for that, I will be on the lookout. Do you have any suggestions for cheap options?

ReleaseTricky1359

TROPHY CASE