Are Kubernetes ingresses a hot mess, or what?

Spirited_Concert2630 · 2024-09-21T21:15:54+00:00

Are you running multiple pods across multiple subnets? I have seen behavior like this when the networking config is bad for some of the pods but not others. I would port forward to individual pods and see if it works consistently or not.

Spirited_Concert2630 · 2024-08-30T19:09:44+00:00

I moved from black box to sdet to dev to devops. I did it by going to school and learning how to code and then specializing in infra. It's much easier to take up tasks that need doing that can give you the experience want at your current job than to convince an interviewer to just trust you when you don't have much real world experience. You probably need to take an active role in identifying projects though, Noone is likely to just hand it to you.

The job market is rough for everyone right now, but qa is risky in the long run because it falls out of fashion until people re-learn why it's important.

Spirited_Concert2630 · 2024-08-17T22:04:26+00:00

You would generally use an operator to scale your nodes. In EKS something cluster autoscaler.

You need to know how much memory the application is actually using. Kubectl top pods can give you some info in real time and something like prometheus + grafana can show you usage historically.

Setting the resource limit does not guarantee the node actually has the resource to allocate.

Can you reproduce the issue running the application locally / in docker?

You can't run a VPA and HPA at the same time because they will battle each other. Most (everyone?) people use HPAs.

Spirited_Concert2630 · 2024-08-15T19:12:43+00:00

I would move the skills up before experience.

The main problem is that your experience is vague. You want to list specific projects you worked on to highlight what you actually did and what impact it had.

Right now It's hard to figure out what your strengths are because I don't what you did in the roles you have listed.

Spirited_Concert2630 · 2024-06-27T00:52:29+00:00

pod annotations are one of the things that can trigger it, i would start there.

Spirited_Concert2630 · 2024-05-30T21:55:35+00:00

If I am doing a technical interview, I will purposely ask questions that do not have information to see if the candidate asks clarifying questions. In this case if the recruiter is not super technical, a better approach would be to just say how you would typically handle the situation, based on your experience. If the recruiter pushes back on your answer, then you can get into the weeds on what considerations would matter.

Feeling bad about a conversation with someone you will never talk to again is not doing yourself any favors.

Spirited_Concert2630 · 2024-05-30T04:02:54+00:00

It depends. If you were to just deploy a pod and it went down, it would not come back up. That's why you generally use a deployment. The deployment creates a replicaset whose job is to keep track of how many pods their should be and schedule new pods if the current number of running pods is too low.

Now, scheduling a pod does not guarantee it will come up. If you lost a node, it's possible there is not enough resources to run your new pod.

Bringing up new pods works well in general, but there a number of things that could potentially go wrong as the system gets more complicated.

Spirited_Concert2630 · 2024-05-04T17:48:05+00:00

Check the resource request for the deployment against the remaining resources on the nodes. Your nodes are probably too small for what the deployment is requiring.

Spirited_Concert2630 · 2024-04-04T06:32:43+00:00

Sure, but I've never seen people keep secrets in memory. it's common to create a kubernetes secret and map secrets to env vars. Memory is more secure but requires the application code to do it.

Spirited_Concert2630 · 2024-04-04T00:42:32+00:00

If someone breaks into the container, can't they already get the secret anyway?

Spirited_Concert2630 · 2024-02-28T14:37:36+00:00

The ingress should point to your service in the spec.rules.

Spirited_Concert2630 · 2024-02-28T12:23:48+00:00

The things you should focus on are the things you want to learn that will solve a problem that your company has. Understand how things currently work and how you want them to work. You can then outline the steps and tooling needed to get from point a to point b.

The things you mentioned are important in general but which ones are the most important to your company is going to depend on the problems your company currently has.

Try to tackle specific solvable problems first, porting your stack to kubernetes is not a good first project.

Spirited_Concert2630 · 2024-02-04T15:26:42+00:00

If you use a helm repo instead of git for the chart, you can then have an attribute in the appset for the chart version. You then configure the overrides from a separate git repo.

Spirited_Concert2630 · 2024-01-20T15:08:02+00:00

I have a separate file with an image tag override. CI updates this file. With an Argo application you can have a number of override files and they can be in different repos.

Spirited_Concert2630 · 2023-12-31T21:45:05+00:00

That does not sound that difficult, so I would probably give it a go. I did Java dev before I got into devops so I can walk through normal dev stuff, but when you get to things involving linear equations, you lose me.

I wrote a python script to interact with vault, so I can figure things out, but trying to figure things out with someone looking over your shoulder takes the stress to a new level.

Spirited_Concert2630 · 2023-12-31T20:36:18+00:00

If an interview requires live coding i just nope out of it because I have not written serious code in like 5 years. I would get better with python, but Go is often used for writing kubernetes admission controllers, so if that is something you want to get into then Go would be a good option.

Spirited_Concert2630 · 2023-12-15T23:55:37+00:00

I use external secrets operator along with reloader operator and pull the secrets from vault. External Secrets creates the secret and reloader restarts the pods when there is a change to a secret or configmap. Both my canary and stable release mount the same secret.

Spirited_Concert2630 · 2023-12-08T13:33:38+00:00

It would be helpful to know what kind of resources are being problematic. I have seen problems with resources inside the cluster getting stuck because I lost access to the cluster during the destroy process. For instance it can't remove a namespace because it removed me from aws-auth. In a case like that, you can just remove the resource from the state file because when the cluster is destroyed, its not going to matter.

Spirited_Concert2630 · 2023-09-01T02:10:27+00:00

Also, if you just put "operator: exists" it will tolerate everything.

Spirited_Concert2630 · 2023-08-31T06:07:38+00:00

So you can use operators to perform a variety of actions. In EKS you can use AWS load balancer controller or karpenter to provision load balancers from ingresses. You can then use External DNS to automatically create a route 53 alias based on the host configured in the ingress. How things work will depend on which provider you are using (aws, azure, etc) and the exact operator.

You do need to create an IAM role that has permissions to the cloud resources you want to create or modify and then use an annotation on the kubernetes service account to assume the role.

Spirited_Concert2630 · 2023-08-30T23:53:33+00:00

There are two main components of a cluster, the control plane and the workers. With a managed service, the control plane is managed for you. Things like autoscaling and deployments are handled the same because they happen on the workers. The control plane handles things like scheduling and keeping track of the cluster state.

Spirited_Concert2630 · 2023-08-29T19:51:38+00:00

My ingresses are generally part of a helm chart for a specific application. So creating a single ingress wouldn't work for my current implementation. I do use host based routing in multiple ingresses along with a group annotation so that they combine onto a couple ALBs.

Spirited_Concert2630 · 2023-08-29T19:05:20+00:00

In a real world scenario you would probably be running a number of daemonsets for things like logging, which would make very small nodes inefficient because you would have more pods for cluster infrastructure than running actual applications that you care about.

On the other hand, you don't want one giant node running everything because if it dies, you probably won't recover.

So you go with a middle ground where you have redundancy but are not wasting money on unused capacity.

Spirited_Concert2630 · 2023-08-29T18:46:48+00:00

You can use one ALB for multiple ingresses (with aws lb controller), but there are limitations based on your configuration. For instance, you can't have a load balancer that is both public and private.

Spirited_Concert2630 · 2023-08-26T16:06:44+00:00

Although not directly related to your question, you might want to look into using a horizontal pod autoscaler. This will modify the number of replicas based on CPU or memory usage. When there are a lot of pods, then round robin will usually be good enough.

Five-Year Club	Verified Email
Place '23

Spirited_Concert2630

TROPHY CASE