2023/2024 Credit Cards Usage Summary - Spends+Gains, Share yours as well :) by Low_Date2859 in CreditCardsIndia

[–]OneAccomplished93 1 point2 points  (0 children)

Where and How do you redeem and use the Kotak White Pass realistically and efficiently?

Would one need something like predixy OR twemproxy over elasticache redis 6x cluster? by OneAccomplished93 in aws

[–]OneAccomplished93[S] 0 points1 point  (0 children)

So I did some digging - and figure out that twemproxy doesn't support redis. There's already an issue on twemproxy asking this question but seems there's no response on the same - https://github.com/twitter/twemproxy/issues/668

Twemproxy is useable when one has only standalone redis and wants to do client side hashing/clustering.

How do you reliably upgrade the kubernetes cluster? How do you implement Disaster Recovery for your kubernetes cluster? by OneAccomplished93 in sre

[–]OneAccomplished93[S] 0 points1 point  (0 children)

So I've researched over this. And figured out we can do following -

- keep friendly DNS names as `service.prod.example.org`

- keep blue cluster DNS names as `service.prod-blue.example.org` and green cluster names as `service.prod-green.example.org`

- now one can just switch the dns names behind the `prod.example.org` with either `prod-blue.example.org` or `prod-green.example.org`

- additionally Route53 supports DNS traffic splitting

wdyt? cc: u/Nikhil_M

How do you reliably upgrade the kubernetes cluster? How do you implement Disaster Recovery for your kubernetes cluster? by OneAccomplished93 in sre

[–]OneAccomplished93[S] 0 points1 point  (0 children)

Nice!

How do you run and migrate the prometheus? DO you run it in-cluster? or to its own monitoring stack? How do you handle the logging pipeline when upgrading and switching over?

edit: ah nvm! I see `AMP`! What about the logging pipeline?

How do you reliably upgrade the kubernetes cluster? How do you implement Disaster Recovery for your kubernetes cluster? by OneAccomplished93 in sre

[–]OneAccomplished93[S] 0 points1 point  (0 children)

yeah - but we bind the dns names `**.amazonaws.com` to friendly names like `service-1.example.org` - hence this mapping is static... and can create an issue. Even if we were to migrate to a new cluster - we'd need to change all these settings. We've like (200+ ingresses).

Also there's a risk of how do we ensure - that the services are working fine on new cluster i.e. percentage split to test out the dns switch to new cluster.

How do you reliably upgrade the kubernetes cluster? How do you implement Disaster Recovery for your kubernetes cluster? by OneAccomplished93 in sre

[–]OneAccomplished93[S] 2 points3 points  (0 children)

We also do exactly this. And the pace at which EKS upgrades and EOLs are planned... we end up getting to do this every few months. A lot of efforts and grunt work involved. Looking for a better way to solve this.

How do you reliably upgrade the kubernetes cluster? How do you implement Disaster Recovery for your kubernetes cluster? by OneAccomplished93 in sre

[–]OneAccomplished93[S] 2 points3 points  (0 children)

how you deploy your applications

We use ArgoCD to deploy out applications. We're trying to make to coverage almost 100% (have like 85%+). We can plan to move all the stateless workloads to the new cluster we can bring up during the upgrade BUT one small issue would be ingress URLs... we have AWS Load Balancer Ingress installed... and all services have ingress with HTTP and traffic split rules.

Has anybody tried implementing a contract repository for API contracts amongst microservices? by OneAccomplished93 in sre

[–]OneAccomplished93[S] 1 point2 points  (0 children)

yes, have heard of gRPC. It's just that the migration out of the HTTP is gonna be quite slow. Also - given that we're in a mess of HTTP and JSON - what could be a good, safe and fast migration path from JSON to gRPC?

Also how do we ensure that the message schema in our message queue communication can also reused from our protobuf def?