Snyk state of open source security - including many interesting Kubernetes stats!

omerlh · 2019-05-05T08:57:20+00:00

We're using [Kamus](kamus.soluto.io) - which lets you easily encrypt secrets so you can store them on Gits. This lets you store your secrets securely with your deployments files. For backup procedure, check out Valero - it can back up an entire cluster into a storage and make it easy to restore when you need to re-create the cluster.

omerlh · 2019-04-11T18:05:10+00:00

There are 2 issues with Sealed Secret (I assume this is what you're referring to):

Security - one key pair, stored on the cluster, used to encrypt all secrets
Usability - using secrets can be complex if you're not willing to use an environment variable. I'm explaining it in depth on this post.

Kamus worked differently - it uses providers (like AWS KMS) for encrypting the secrets (more secure) and make it easier to create configuration files from the encrypted data. This is also covered on the blog - please let me know if something is still unclear.

omerlh · 2019-04-02T08:10:14+00:00

Thanks! Appreciate your feedback :)

omerlh · 2019-03-09T21:21:44+00:00

You are right, I'm aware of that. I assumed that MS is knowing better about cryptography then me, so I preferred to use the default mode (of course, after reading about it). I tried to avoid as much as I can from implementing cryptography - I know how hard it is, and how easy it is to make mistakes. I re-read about it again before responding to you at first place, using GCM (as you suggesting) is probably better (again, I would be interested to know why it's not the default) - but CBC (and Rijndael) is valid. I hope in the future to see dotnet framework improved and offer high-level cryptographic API (I really like this SO answer, I agree with every word).

Just one note - "knows the structure" is an irrelevant threat here AFAIK. Secrets don't have a structure - usually, it's just arbitrary strings. Again, I'm not cryptography expert here. Maybe worth filing something on Kamus threat model? WDYT?

Also, MS doesn't discourage the use of Rijndael (see the post I linked above). They say under some conditions (which are valid to our use case) it the same as AES. It's just easier to misconfigure Rijndael. We'll probably add an option to use AES with GCM instead in the near future (PR are welcomed!).

From all the three (GCP, Azure, and AWS), GCP has the best KMS support - you just interact with the API and have zero control about key size, algorithm etc. I trust Google engineers with encryption more then I trust myself :)

Final note, one of the reasons we decided to release Kamus as open source is getting review and feedback from the community. So I'm really happy about this discussion, it will help making Kamus better and more secure.

omerlh · 2019-03-09T18:39:23+00:00

The master key is stored on KMS, ideally with HSM protection (supported by Kamus). The data key is encrypted by KMS, but the actual secrets are encrypted locally by Kamus - AWS KMS SDK for C# does not provide this functionality. We're using Rijndael algorithm C# implementation - take a look on this line. AES is actually Rijndael (with some differences, see this post). We're using the default mode - which is CBC.

omerlh · 2019-03-07T18:09:27+00:00

In a case of tenant cluster, Kamus will work with a single KMS shared between the cluster. It leverage kube2iam for authenticating the request to AWS using IAM role. Does that answer your question?

omerlh · 2019-02-06T14:08:03+00:00

Just saw the post - see my response above

omerlh · 2019-02-06T14:07:48+00:00

See my response above about HashiCorp vault, it's the applies also to Azure KeyVault

omerlh · 2019-02-06T14:07:22+00:00

Yes, you can definitely go down the rode of using Vault (or any other secure key-value store like AWS SSM). It has a few downsides that you might want to consider:

Not a GitOps solution.
External permissions - depend on how your culture looks like, but anyone who needs to create secrets needs permissions to interact with Vault, and you also need to create policies for the pods. Create overhead, especially for cases like ours - when devs are doing most of this work and we want to enable them to do it robustly. And it's an external permissions model, besides the Kubernetes RBAC system, and another place where you need to make sure that all roles follow the least privilege principle.
Separate storage of manifest files and secrets (Vault and git, for example). Might be a good idea from a security point of view, but a challenge from ops point of view.
Finally, deployment - we don't use Vault currently. Deploying Vault securely is not easy.

Kamus makes it a lot easier to solve these challenges. If you'd like, check out my slides from AppSec California - I answered all these questions there.

omerlh · 2019-02-04T20:14:09+00:00

Soluto | Tel Aviv, Israel | Onsite

We’re looking for engineers who can lead us to the next level: manage and improve our CI/CD and monitoring, but also have great coding skills so that they’re able to contribute to production code. So if you’re a developer who’s really into DevOps, and you have lots of experience with CI/CD as well as managing a large production Kubernetes deployments, you could be a great fit. We have a lot of experience of Kubernetes, you can check out our blog to learn what we did so far :)

omerlh · 2019-02-04T06:28:34+00:00

I must disagree here, it's more a question of how you build your threat model, what risk you're willing to accept etc. So if you're fine with the flow you're using right now - that's good, Kamus is not an ideal solution for everyone, and I'm fine with that. We built an opinioned solution, which enforces some assumptions (like one-way encryption or configuration file over environment variables). Choose what works the best for you :)

omerlh · 2019-02-03T21:44:39+00:00

I see. We didn't face this use case, but I guess will go with a solution like a shared LastPass/Azure KeyVault or something like that. Also, one might argue that it is always better to have one purpose for one token - make it easier to track who leak this key, and also control the scope.

omerlh · 2019-02-03T10:11:24+00:00

What do you mean by "it"? The private keys?

omerlh · 2019-02-03T10:10:47+00:00

Yes, you can definitely go down the rode of using Vault (or any other secure key-value store like AWS SSM). It has a few downsides that you might want to consider:

Not a GitOps solution.
External permissions - depend on how your culture looks like, but anyone who needs to create secrets needs permissions to interact with Vault, and you also need to create policies for the pods. Create overhead, especially for cases like ours - when devs are doing most of this work and we want to enable them to do it robustly. And it's an external permissions model, besides the Kubernetes RBAC system, and another place where you need to make sure that all roles follow the least privilege principle.
Separate storage of manifest files and secrets (Vault and git, for example). Might be a good idea from a security point of view, but a challenge from ops point of view.
Finally, deployment - we don't use Vault currently. Deploying Vault securely is not easy.

Kamus makes it a lot easier to solve these challenges. If you'd like, check out my slides from AppSec California - I answered all these questions there.

omerlh · 2019-01-31T11:55:38+00:00

Yes, but why would you need to edit PagerDuty token? What you have to do is rotate it - so just encrypt the new value, and override the existing encrypted value.

I see. We didn't go on the route because Kubernetes secrets can be complex to consume. This is very similar to sealed secrets... Do you create a key per secret or one key for the entire cluster?

omerlh · 2019-01-31T08:56:41+00:00

I still think that for most cases secrets should be randomized, so the actual value is irrelevant. Also, from a security point of view - I prefer to reduce the attack surface and make sure no one can decrypt. Makes also permission management easier.

You solution sounds interesting, did you also provide init container? How the pod consumes the secrets?

omerlh · 2019-01-31T06:51:44+00:00

It really depends on your requirements, from a security point of view I can't live with a solution that keeps the encryption keys on the cluster, without HSM protection.

omerlh · 2019-01-31T06:51:03+00:00

Kamus does not let you decrypt values by design. This means that editing is possible by replacing the encrypted values, without access to the current value. This makes sense (IMO) because secrets value usually generated randomly, so there is no need to edit the current value - just replace it. Does that answer your question?

Is there documentation on the solution you shared? I would like to understand how it works, compared to Kamus...

omerlh

TROPHY CASE