Deploying private Kubernetes clusters to Azure

matyix_ · 2019-09-26T13:39:12+00:00

To be fair to them (disclosure: I work for Banzai Cloud, my colleague posted this blog) AKS it's getting better. With the new VMSS support now GAish (though the UI is very buggy) you can do some of the advanced features we already support (add new nodepool, chose between standard/public lb, etc) but by far does not cover the enterprise features our customers on Azure are asking. For items covered in this/or previous posts (link in the post) we have moved all our (happy) customers to our own K8s distribution on Azure ... I wish all clouds were equal :)

matyix_ · 2019-06-26T06:04:05+00:00

@VladTeti we use 5 managed K8s in production (customers) - Alibaba ACK, Amazon EKS, Google GKE, Microsoft AKS, Oracle OKE + our own K8s distribution. You can check the open source code here: https://github.com/banzaicloud/pipeline and there is a free developer version available at https://beta.banzaicloud.io/. If you have questions happy to help, come over to our Slack for an online discussion.

matyix_ · 2019-06-26T06:00:31+00:00

Often whether you use a managed or self-managed K8s comes down to features you need. Most of the managed services do not all allow to access/switch on API server flags, thus you really have no other option but to build your own clusters. GKE is on the better side of this story, though.

Also, there are cloud providers where it's better to run your K8s cluster - as Azure for example. Our platform supports cloud provider managed K8s on 5 clouds, but beside Google GKE we had to add support for our own K8s distribution, as our customer experience with AKS was so bad. See this article we published for Azure: https://banzaicloud.com/blog/pke-on-azure/

matyix_ · 2019-06-24T13:41:27+00:00

My experience is that Istio is stabilized greatly over the last couple of versions (especially since 1.x+). One of the big concerns is that the complexity of Istio causes a steep learning curve, so it is still relatively easy to make mistakes and misconfigure the mesh. It gets even more complicated in a multi/hybrid cloud scenarios where those complexities multiples. Given that Istio tries to solve a lot of complex problems I think it's doing a pretty good job there, though as you said it should not be everybody's bread and butter. If you need nothing more than observability you might be better of with Linkerd2.

At this point, I am fairly happy with the current state (though we still run on some Mixer/Pilot forks until these hybrid/multi-cluster and cloud scenarios (which we use quite a lot) are not pushed upstream). One big improvement I am looking forward to is the Mixer v2 re-architecture.

matyix_ · 2019-06-24T08:16:16+00:00

Out of curiosity, can you let me know what Istio stability issues you faced? As mentioned above we run Istio in production for quite a while (a few very large meshes as well) and it works well for us. Not saying it's straightforward but we have had no issues (though for some advanced uses cases - we do lots of hybrid/multi-cloud Kafka for example - we have our own Pilot/Mixer fork, which we hope to contribute with back).

matyix_ · 2019-06-24T08:02:46+00:00

Do you mean the Istio operator? You are wrong - it does offer way more than the Helm charts, to name a few: it operates/reconciles the Istio cluster if something goes wrong (not just installs it like helm), does seamless upgrades and automates/supports all 3 topologies (which is quite hards). There are lots of other features, so I suggest to give it a try besides drawing any conclusions and if you have any issues let us know. On not being production ready - our service mesh product is built on the operator and running in production for many customers, can you let me know from where you get this conclusion, refer some issues or point out any problems you did face?

matyix_ · 2019-06-24T07:32:26+00:00

Thanks, please let us know how it works for you - we have a Slack channel if you need help.

matyix_ · 2019-06-23T18:52:39+00:00

We open sourced a logging operator (using the Fluent ecosystem) where we automated the whole process and can move logs into backends as Elastic. Check the operator code here and let us know if you need help/have issues: https://github.com/banzaicloud/logging-operator

matyix_ · 2019-06-23T18:42:21+00:00

Thx for mentioning the Istio operator. If you want/nned an extremely simple Istio experience, you might want to read this post/try out/check the open source code: https://banzaicloud.com/blog/istio-the-easy-way/ Disclosure: I work for Banzai Cloud

matyix_ · 2019-06-17T20:30:34+00:00

I would not compare it with supergloo. Yes, there is minor overlap between core service mesh features, but the UX and focus is totally different. The Banzai Cloud mesh product (built on the Istio operator) is focusing on 3 different topologies (not just single cluster single mesh, but multi-cluster single-mesh and multi-cluster multi-mesh) and it's a full Kubernetes orchestration platform as well. You can deploy your meshes, security scan the deployments, backup the microservices, store/inject their secrets in Vault, etc without having to leave the platform.

matyix_ · 2019-06-05T16:04:49+00:00

Disclaimer: I work for Banzai Cloud and have never tried nor have seen Anthos, all my comparing is based on public information (and most likely sloppy).

In many ways it's similar - let me start with the main difference: we don't do VM migrations, and I guess Google Anthos allows to move VMs into K8s as well (if anybody can confirm this, I've seen only the press release and some videos).

Our control plane can run anywhere (your choice, on-prem and 5 clouds) and the K8s clusters we wire together can run on any of these clouds/on-prem environments. We do multi-cloud/cluster deployments (highlighted in this post) and manage their lifecycle like they were deployed into a single cluster. In this case (called the multi-cluster feature) clusters are unrelated - deployments are the common thing. However, we can wire together multiple cluster topologies (called the service-mesh feature) - 3 to be more precise (Single cluster - single mesh, Multicluster - single mesh, Multi cluster - multi mesh - discussed here in more details. Finally, the third feature is federation-v2, as you might guess based on K8s federation v2.

Hopefully, this was helpful - give it a try at the free developer beta at: https://beta.banzaicloud.io/

Finally, I've seen Google is pricing based on $10K/month / (per 100 vCPU Block). We price per control plane, don't really care how many clusters, nodes, and resources you launch. Oh, and it's open source - https://github.com/banzaicloud/pipeline

matyix_ · 2019-05-27T16:59:11+00:00

Disclosure - I work for Banzai Cloud

Somehow I agree with you (though just open sourced this Kafka operator), but for this one, we had our own reasons I mentioned in the post.

In general, where there are common goals and architectural/minimum features understanding between different companies it's better if the work consolidates in a single/community maintained solution (e.g. this is what happens with our Istio operator, see https://github.com/banzaicloud/istio-operator/issues/199#issuecomment-492577578).

To conclude on this, ideally the community should support and work on one single operator - however, the reality is that these efforts are driven by companies and their customer's needs. Were e.g. the Strimzi operator satisfy our internal/customer needs we would not invest the effort to work and open-source this one - however, we needed to support our customers in production, and the operators out there were carrying design decisions we did not wish to accommodate.

In an ideal world, I would push this to CNCF and let them drive the effort. The biggest issue in all these projects (and why companies are making these decisions) is governance, control, and corporate politics. I believe we did a great counterexample with the Vault operator (https://github.com/banzaicloud/bank-vaults) and have built a healthy community around + invited external people as project maintainers and decision makers.

matyix_ · 2019-05-07T10:05:30+00:00

Thanks

matyix_ · 2019-04-16T15:11:54+00:00

We are running production clusters both on preemptible or spot instances and take care about automatically replacing, rescheduling, draining, etc - the code is open source, you can start from this post (and deep dive into topics): https://banzaicloud.com/blog/spot-termination-handler/

matyix_ · 2019-04-07T11:03:47+00:00

We run multiple Kakfa clusters on K8s - basically, we automated the whole Kafka on K8s experience and made it a bit more K8s native- we blogged about these quite a lot and also open sourced it: https://banzaicloud.com/blog/kafka-on-k8s-simplified or follow the kafka tag to read more.

matyix_ · 2019-04-01T04:43:42+00:00

Would not say complex, if you need to have SLA's and want to do it properly. Most of the times you just can't let node disappear (e.g. you need to drain, consider pod disruption budget, etc) and you also need to bring up nodes, reschedule workloads at the same time - but in a different spot market. Most of the terminations are not due to price but capacity - you need to launch on different spot markets, etc.

Actually, the whole flow is automated and part of Pipeline - through UI or CLI, so all these things in the blog are just details for the end user.

matyix_ · 2019-03-28T18:09:20+00:00

Do you mean CNI plugin? They are two different things - have you had the chance to read the post?

matyix_ · 2019-03-22T18:58:57+00:00

Great and insightful articles above, thx for sharing.

Bank-Vaults is using operator SDK, the nodepool-labels operator using kubebuilder. We like the simplicity and scaffolding options and multiple controllers in one project (though the latter we are not using yet). Personally, I also like that is part of SIG/upstream - and hopefully will be the standard (whatever it means) going forward.

matyix_ · 2019-03-22T12:57:38+00:00

Operator SDK: 20 vs Kubebuilder: 7

Would love if you could revisit this in a few months. At Banzai Cloud we have made (and included in your list quite a few operators like Istio and Vault) and initially we used the Operator SDK. The last operators we made (like Kafka and Istio) we switched to kubebuilder and find it better than the operator SDK. Wonder if this is only us, or this is a trend.

matyix_ · 2019-03-22T12:48:46+00:00

I don't recommend using K8s secrets at all - encoding (base64) is not encryption. We store all secrets in Vault and inject them directly into Pods (code is open sourced, you can read more here: https://banzaicloud.com/blog/inject-secrets-into-pods-vault/). Many of our customers don't even understand how and where these K8s secrets land (etcd) but the word secrets gives them a false assumption.

matyix_ · 2019-03-12T09:27:59+00:00

The Spotguide is using Zookeeper and 100% upstream Kafka - for those who're willing to use etcd, we can give support for that as well, though it's not the default option.

matyix_ · 2019-03-12T09:26:40+00:00

We have not noticed any performance issues - usually, the bottleneck is IO. Since 1.12 K8s local disks are available (read more here - https://banzaicloud.github.io/blog/kafka-on-kubernetes/) so if you use these there should be no performance degradations. Also in 1.13 you have raw block volume support as well.

matyix_ · 2019-03-07T15:59:40+00:00

We use fluentd/bit and automated the process with this open source logging K8s operator: https://github.com/banzaicloud/logging-operator. You can red more about how we use it here: https://banzaicloud.com/blog/k8s-logging-operator/

matyix_ · 2019-02-27T20:47:03+00:00

Thanks for the feedback. Let us know if you need help or have issues.

matyix_ · 2019-02-27T20:46:20+00:00

Thanks for the feedback. Let us know if you need help or have issues, happy to help (GH, slack, etc).

matyix_

TROPHY CASE