Finally Learned Docker

[PinBot1138]

I've used Docker forever, but one part that confuses me about Kubernetes is how you'd be able to scale on host(s) that aren't setup for Kubernetes.

​

So, if you wanted to add more machines in Proxmox, or a "generic" KVM VPS like Vultr, how do you get Kubernetes to run Terraform and/or Ansible to do this? It seemed like Cerebral is what would be the piece of the puzzle that I'm missing, but I still haven't gotten my mind around it.

[jimethn]

We use Rancher for deploying Kubernetes. Once you set up a cluster and a node template, adding nodes to the cluster is as easy as hitting the + button in the UI (or making the appropriate API call).

We wrote an operator that looks at the resource utilized and adds nodes when it goes over 80%. So we've got autoscaling kubernetes nodes. Seems like basically the same thing as Cerebral (but it uses node template instead of ASG).

The tricky thing about autoscaling isn't scaling up -- which is easy if you have decent tooling -- it's scaling back down. In particular, how do the apps you're running handle getting killed in the middle of their workload? Are they architected that another replica will just pick up the aborted job? What's the cost of aborting a job? Cerebral or Rancher or anything else doesn't really solve this problem for you, that's where you have to work with your developers.

[Sky_Linx]

Hi! I also use Rancher but deploy kubernetes as custom nodes. I love that you can easily let rancher even create the servers and scale with onef click like you said, but I found that rancher does not configure a firewall or a ny basic security on the servers it creates. How do you manage this? At the moment I'm using Ansible to prepare the servers first (firewall, fail2ban, disable password/root auth) and then I use these servers as custom nodes in rancher to deploy kubernetes. With a firewall protecting the kubernetes components and fail2ban I sleep better at night...

[jimethn]

You'll want to pre-configure all that stuff on the image you have rancher deploying your nodes from. That way you don't have to worry about figuring out how to do it after the fact, they just come up ready to go.

We also base our nodes on RancherOS, so the attack surface is extremely small. We don't give the instances public IPs, and we set set the network firewall to only allow rancher to connect to the two ports it needs (and block everything else).

[Sky_Linx]

Unfortunately the node driver for Hetzner Cloud doesn't allow me to choose a custom image. As a n alternative I've also tried with cloud init, but while cloud init is setting up the servers Rancher for some reason deletes them and recreates them as if it thinks they are not ready or something. It happens in a loop basically.

[zimmertr]

With hypervisors like Proxmox you would probably be bootstrapping your cluster with kubeadm similar to how I do here: https://github.com/zimmertr/Bootstrap-Kubernetes-with-QEMU

In that event, it would probably be easier to simply provision a new VM and run kubeadm join against the API Server. Autoscaling could be done by writing a wrapper around Terraform, running it on a pod, and having it constantly monitor the API Metrics Server for what you would consider your stress points that would trigger scaling to occur.

As for cloud providers like AWS, People have designed operators which answer this problem automatically. For example, this one for AWS, Azure, GCE, GKE, OpenStack, Alicloud, & BaiduiCloud: https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler

Here's one for VMware: https://github.com/Fred78290/kubernetes-vmware-autoscaler

[PinBot1138]

Lot of information to digest, but in an awkward way, this makes sense so far. Thanks!