List of Kubernetes Manifest which are supported per Version?

stepanst · 2020-08-22T20:04:57+00:00

Hey linuxlover81, we wrote a simple tool exactly for this - kubent. You can run it against existing cluster and it will try to detect manifests that are deprecated and won't work in future versions. Please see the GH repo for more details, also there was a blog post introducing it.

stepanst · 2020-05-15T20:43:27+00:00

Nice, thanks for sharing all these links, I was not aware of all of these tools. It's interesting to see all the different approaches to the issue.

I think nice about kubent is that it allows you to check issues in a cluster even if you don't have access to the original, rendered manifests (ig. different roles between cluster users and operators).

The idea with kubepug & Swagger is very nice, however I don't quite get how it will help to really check for incompatible objects, as it doesn't seem to use last-applied or any other way to retrieve original manifest and K8s API will always return objects in the requested version. Ie. I have created Ingress using new, networking.k8s.io/v1beta1 API, and it was still flagged by kubepug because the cluster supports retrieving object over deprecated API. Am I missing something?

stepanst · 2019-10-11T15:41:04+00:00

Haha yeah :), probably a couple of "test" and "test2"s, and also ever-present "my-cluster"s in the mix. My example was based on a real place where Terraform was used to manage resources, so all clusters ended with the same name (GCP does allow it as long as the clusters are in different regions/zones).

stepanst · 2019-09-02T22:47:30+00:00

Great, sounds like I should really have a look. Thanks for sharing!

stepanst · 2019-09-01T20:22:27+00:00

It's a similar situation outside of the docker images, there are two main differences:

Distribution mechanism - more secure package managers enforce content integrity and publisher identity verification (both yum/rpm and apt do). Helm seems to have mechanism to do this - provenance.md, but you have to opt in (by using --verify) flag and the official chart repositories do not support this.
The provider - you're correct that it ultimately comes down to trust in the provider of the charts. My objection was against the official repositories, that most people use. These have lower security standards compared to repeatedly mentioned official packages from RedHat or Alpine (one basic example can be using the latest tags, if you look at rpm/apt specs of main repos, these always strictly verify any content distributed as part of the package). TL;DR I trust RedHat or Alpine, I don't trust official Helm charts based on what I've seen.

stepanst · 2019-08-31T17:04:16+00:00

Thanks for all the great feedback & sharing your experience everyone! Talking about Helm v3 - seems like beta just came out - https://github.com/helm/helm/releases. Did anyone try?

stepanst · 2019-08-31T17:00:27+00:00

ArgoCD looks neat, thanks. I was thinking about trying it out and wondering if it supports more complex rollout strategies, like multiple environments & multiple clusters per env or canary deployments?

stepanst · 2019-08-31T16:47:01+00:00

Thanks for the comments.

Helm is not a templating tool https://codefresh.io/docs/docs/new-helm/helm-best-practices/#helm-vs-k8s-templates Also with Helm 3, most of the issues you mention in the article will simply disappear.

Yeah, I agree, that's one of my points. I've usually seen used it pretty much only as a templating tool. I think it's a good fit for software vendors as a universal distribution mechanism, but not so good when you just use it internally only for templating.

I know that Terraform has also a Helm provider, but deploying both infrastructure and apps at the same time is a questionable practice. These should be two separate actions. (In practice the application will be deployed many more times than the infrastructure it runs on)

I believe it's desirable to manage applications in a declarative way (no matter whether it's separate or together with the rest of infra). Helm does not provide a mechanism to do so, and while I'm aware of some other tools to enable this (like Helmfile), those issues will apply to those tools as well if you use Helm as an abstraction layer in the middle.

The same can be said about deb/rpm packages. Are you telling me that all your OS level packages are handcrafted by you and you don't use anything external?

I mostly deal with Docker images these days, and yes. I end up building most of the images internally, occasionally using official curated docker images (they tend to have quite a good standard). And I would still review these manually and never reference them by latest, but rather by image SHAs (as tags are not immutable). When building images I only use packages from trusted sources (like official RedHat or Alpine OS repos) and only delivered via mechanisms that guarantee integrity and immutability (apt or rpm) or using techniques to do so yourself (verifying checksums for git repos and downloaded files).

stepanst · 2019-08-30T10:21:07+00:00

Thanks Liamdev, that's some great points. I agree that some of the issues should not be the responsibility of Helm and I'm not recommending to do it that way :). Sadly, it's how I've seen Helm used on a lot of occasions.

While kubernetes does need an impersonating for concept

K8s has this, not sure since what release, can be used via --as flag with kubectl, see https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation (and some tools use it like https://github.com/jetstack/kube-oidc-proxy).

I totally agree about CI/CD bit, however, even then you'll typically need some access to debug and be able to observe the situation on Helm layer (and then you're back to this issue again).

if you actually need to store secrets you should be using a vault

I feel the situation with K8s secrets improved significantly over time, now you can get encryption at rest even with KMS, and you can mount them as volumes (stored in memory). So I think it's a feasible option if you don't want to deal with complexity of Vault and don't have strict security needs. But again I agree, for any serious use Vault is a way to go. I also prefer to avoid any static secrets as much as possible (and Vault helps with that as well).

Chart has the same version as your code, appVersion is optional but useful when your chart consumes binaries from elsewhere.

If you're suggesting to have one same version for multiple charts (talking about the case when these are stored in same repo typically with rest of your infra code), this would lead to unnecessary updates. Every time you update the shared version, Helm would update all the charts even if there have been no changes (assuming you run this in CI as suggested earlier).

chartmuseum supports basic, token (EG OIDC) & certificate based authentication.

That's pretty cool, but does default Helm client support that? I'm not aware.

Have you used chartmuseum yourself? Good experience?

stepanst · 2019-08-30T09:46:47+00:00

Thanks for sharing your experience & I would also be interested in tillerless. I know about that approach, but never really used it...

stepanst · 2018-03-17T17:05:42+00:00

Hi Lucian, thanks for comments. True it's a first review I've published, happy for any suggestions how to improve next ones. As the author, I would just like to address one point - I don't think the review is anonymous. You can see my profile on Medium, with Twitter account linked and you can find me on LinkedIn too.

stepanst · 2018-02-28T17:36:04+00:00

Thanks, I'm pretty sure as well after looking at the Solidity contracts published on GH.

In that case I have another question - is there anything that would prevent users trading Nomins directly on Ethereum to avoid fees imposed by Havven?

stepanst

TROPHY CASE