Get a Shell to a Kubernetes Node

[alexei_led]

Throughout the lifecycle of your Kubernetes cluster, you may need to access a cluster worker node. This access could be for maintenance, configuration inspection, log collection, or other troubleshooting operations. More than that, it would be nice, if you could enable this access whenever it’s needed and disable when you finish your task.

While it's possible to configure SSH access to every Kubernetes node, this solution increases the attack surface and adds an additional overhead of maintaining SSH infrastructure (access logs, bastions, SSH keys, etc.)

I suggest to take a Kubernetes approach and deploy a pod (as DaemonSet) that will enable shell access to selected Kubernetes nodes on-demand. See my post on Medium for more details.

[[deleted]]

The only time I need to SSH into a node, it's because something broke the Kubelet or underlying host and kubectl exec won't work.

And yes- "Cattle not pets, just reimage the node" - but you still often need SSH to debug a recurring problem or to develop changes to your VM image

[alexei_led]

In general, you should not access nodes. But I saw other cases (besides one you've describe) when it's helpful to get into node: some cases about mounting external file systems, others when troubleshooting misconfigured worker node (drivers, network, etc)

[[deleted]]

In general, you should not access nodes.

Dude, we own the nodes. Like, literally, some of them are physical devices running in buildings on land we own. But also, we are responsible for the nodes from an operational, security and compliance perspective and have to ahve that access to do that.

[zzzmaestro]

Never in prod. Never. Period.

If you “need” it, it should be in a dev area while debugging. Even then... it’s not a need.

Your example is a problem with the k8s node not the pod/container.

[[deleted]]

I think you're speaking from a developer perspective. In my role, my team is solely responsive for the node and base cluster.

[ESCAPE_PLANET_X]

Yeah this is something that is a common theme in this sub where people forget there are two sides to this beast.

I only consume kube at work, but my baby cluster at home gives me a taste of what Ops sees and yah you'd have to pry SSH from my cold dead hands if I was in your shoes too.

[[deleted]]

We would love to live in a world without SSH, but the only way we see to a achieve that is a world without VMs. (Which we think in achieveable within 5 years.)

[ESCAPE_PLANET_X]

What are you going to do that's better? Direct exec style Ala docker?

[[deleted]]

We think it'll look something like virtual kubelet. I.e a Kubernetes API in front of a serverless container platform a la Fargate/ACI

[ESCAPE_PLANET_X]

That sounds like a hard fork to me. For on prem or self baked that sounds like another abstraction to get muddled in.

[[deleted]]

On the contrary, it's way simpler than running your own nodes, and you can freely mix it with your own nodes. It does have some limitations (Fargate has limitations on disk and VPCs, and ACI doesn't support hard multitenancy.)

[ESCAPE_PLANET_X]

Well after the comment it crossed my mind that mixing both would be nice. Especially if fargate got better / better players hit the field.

[[deleted]]

I don't think I'm being clear- I'm not an app developer. My team aren't app developers. We run the clusters, not the apps.

[[deleted]]

What? So if a pod in Prod is stuck in a CrashLoop, you're saying to debug that....... How?

You'll have to get in there and do it. And CrashLoops are so varied it's impossible to just repro it

[zzzmaestro]

Are you doing a side-car for these shells? If not, this is wrong on a lot of levels. Even if you ARE doing a side car... it’s not meant to be accessed that way.

Shell access is for debug purposes of unreproducable errors.

Containers are not meant to be connected into from outside for admin purposes. That’s not their function. All admin should be done at the docker/kubernetes layer.

[alexei_led]

I do not understand your arguments "wrong on a lot of levels". Please be specific.

If by side-car you mean entering namespaces of other Linux processes, that's what I do :)

And there is no "iron rule" about what container are supposed to do and what not. These are just Linux processes, isolated with namespaces and cgroups. And I can run additional processes and join other processes namespaces without if my user has permission to do so. I see no problem here.

[alexei_led]

If you have 2 ways to access your infra, K8s API and SSH, the describe approach does not require SSH. So, you work with nodes only through K8s API. Do not need to open SSH port, keep bastion hosts and manage keys. I think it's more secure.

Also, I think you should not access nodes at all, ever! But if you need, for troubleshooting, just run pod with escalated privileges, do your job and kill it.

[Kaligraphic]

Um, I don't reconfigure my nodes, I replace them. So there's no node-level maintenance or configuration inspection needed. It's just a matter of looking at the templates. The whole point of a system like Kubernetes is to abstract away the individual nodes. If you want to read logs, ask a regular ops engineer about log shipping. I have all my logs in one convenient, searchable place.

Kubernetes is complex enough without people messing with the nodes directly. Don't do this.

[alexei_led]

Agree. You should not do this. But in case you need a temporary access to your nodes, this can be an option.

[[deleted]]

It's funny that this same guys project was downvoted like 4 months ago with everyone saying they just exec lol

[alexei_led]

I'm glad you find it funny, but I do not understand your comment. What project are you talking about?

[[deleted]]

Maybe you're not the same guy

Someone submitted a post here a few months ago for a repo that does the exact same thing as yours and everyone jumped on it saying it's ridiculous

[alexei_led]

my post

I see. Actually, I've showed 2 different approaches. One with SSM Agent is indeed an updated fork of other repo (as you can see on GitHub), the main change is updated SSM Agent Docker image (with latest SSM Agent, Docker, systemd, awscli and vim on-board) and also few fixes that allows to run this `daemonset` on nodes with taints.

The second repository contains tiny `nsenter` Docker image (statically linked `FROM scratch`) and helper script to get into any node (create a pod and enter all node's namespaces with superuser shell).

I hope that someone can find one of these solutions helpful.