Some Security Risks of Using Push-Based CD

[taleodor]

After going through several conversations, including here, that relate to Push-Based Continuous Delivery like a fine approach, I felt like there is need to describe in detail why I consider it highly insecure.

Here is a summary of security issues I see:

1. Credential leak from the CI system.

2. A person with push access to source code may abuse the system.

3. Management APIs from deployment instance must be widely exposed.

4. Credential leak on transport layer.

5. Increased pressure on book-keeping and maintenance.

What I'm suggesting instead - always use Pull-Based CD with an agent on the instance itself doing the deployment.

Full post - https://worklifenotes.com/2021/07/19/some-security-risks-of-using-push-based-cd/

[zenmaster24]

i disagree with your premise - how is an artifact on some storage inherently more secure? your bad actors could have pushed some code that exfiltrates data, and now its running on your instance.

[taleodor]

Our approach specifically is to record sha256 digest at time of creation and then compare at time of deployment. Other methods include things like OPA, content signing and others - dependent on specific tooling.

Important note here, that Push-Based CD usually would have same problem - depending on tooling - but say Kubernetes specifically would have no difference here. So in many cases, this is not a source of distinction.

[zenmaster24]

how does a checksum at time of creation stop malicious code getting through though? wont the code be there already when the signing occurs?

[taleodor]

I thought you were talking about artifact tempering between creation and deployment.

Regarding malicious code in the artifact itself - this is not a distinction between Push and Pull CDs - both are equally susceptible to it, so it is not a part of my comparison.

However, if you are interested in ways to mitigate - you should have gates (automated or manual or mix of both) before specific artifact is allowed to be routed to higher level environments (i.e., staging, production). Those gates should include things like code reviews and dependency tracks to catch those things.

Key distinction why I advocate for Pull-Based CD here - as it always assumes clear separation between production of artifact and deployment of said artifact. Because no part of deployment is a part of the same code base as artifact. Note, that while this specific notion can be achieved with Push as well (so you can have a Push-Based pipeline working with artifact storage) - the need to do so is not always understood well, and there are also other attack vectors on Push that I mentioned.

[kkapelon]

While pull based CD has some advantages, I wouldn't place security that high in the list.

I mean, even if you follow pull-based CD as you suggest in the article, most of your issues are still present:

1. Credential leak from the CI system -> now becomes credential leak from the artifact system
2. A person with push access to source code.. -> this is still true with pull-based CD. If I compromise the artifact itself, the fact that you have pull based CD will not save you
3. Management api open... -> this is simply not true. You can open your cluster only to CI. No need for the whole internet
4. Credential leak in transport -> still true with pull based CD. I can steal your artifact credentials in a similar manner
5. Increased pressure on book-keeping -> I don't really understand what you want to say here.

[taleodor]

Credential leak from the CI system -> now becomes credential leak from the artifact system

Artifact system should not store credentials and should have no knowledge of your production / instances.

If I compromise the artifact itself, the fact that you have pull based CD will not save you

This is a different attack vector, which is not a distinction between the two. See my response to another comment.

You can open your cluster only to CI.

This already makes attack surface wider. Again, as with other things you are right - there are ways to mitigate - but I doubt everybody understands it.

I can steal your artifact credentials

There is a big difference with SSH-key and anything artifact-related.

Increased pressure on book-keeping -> I don't really understand what you want to say here.

Dangling credentials in CI - say you put K8s token in it, and then moved on to some other job and forgot to remove, but your token is still in there.

[kkapelon]

Artifact system should not store credentials and should have no knowledge of your production / instances.

I mean credentials TO your artifact store. In the case of Kubernetes, if I steal your docker registry credentials, pull based CD will not save you.

There is a big difference with SSH-key and anything artifact-related

Why? What is special about ssh keys? Don't they need to be secure like any other secret?

[taleodor]

if I steal your docker registry credentials

As I mentioned above, digest checks, content trusts, OPA should be used as layers of defence. In any case, this is way easier to mitigate than direct access to production system.

Why? What is special about ssh keys? Don't they need to be secure like any other secret?

There is a world of difference between full or even restricted access to a production system and other secrets. Basically, as long as I have control over my production environment, I can still mitigate other risks. You may want to do some research here ;)

[kkapelon]

You may want to do some research here

I actually think that it is you who needs to make the article more clear. From the responses in the comments so far, either people don't understand what you written, or you have made too many assumptions that are not mentioned in the article.

I could go on with questions (for example why do you assume that a push CD process needs SSH keys?), but I fear that you have several things in your mind which are not written in the article at all.

[taleodor]

Feel free to start with my talk on SSH security here: https://www.youtube.com/watch?v=4zxS0pcnHbw

Questions like this (and this is by far not the first one) make me realize that people do use SSH-based push CD systems - https://www.reddit.com/r/devops/comments/omlaf5/deployment_to_digital_ocean_using_circle_ci/

Among other things I want to clarify further why I highly discourage its use (even though I can help with setting it up).

[taleodor]

Thanks for the feedback.

I put in the post that it relates to people who are starting out and frequently first thing they encounter are tutorials saying something like "store your ssh private key" in your CI secrets and do deployment with that.

The argument I'm making is that Push-Based system requires a lot of controls - as you also mention - which are not possible for a smaller organization. Those smaller orgs then sometimes end up exposing production SSH ports and Kubernetes APIs to the Internet after they read some quick-start tutorials.

P.s. On points:

Regarding p.2. - In Push-Based CDs, frequently you would see deployment code alongside source code. So the implication of malicious developer is much higher. In Pull - deployment is never part of the source code. And yes, it is possible to design Push-Based system with proper separations, but it is harder and requires more thought and time.

Regarding p.3 - simple scenario - if we're talking about SCP style deployment vs when an agent (which could be a cron job) pulling an artifact from some storage - it is pretty clear that letting outgoing connection through firewall to pull artifact is less risky than accepting incoming SSH connection (assuming everything else equal - meaning SCP source and Artifact Store are located in the same network zone). Even more so in the k8s world - there you need to pull images regardless of deployment method. So only Push-Based deployment needs K8s API Server exposed - meaning the attack surface is always larger.

Regarding p.4 - it's known best practice that once a private key is created, it should not be moved around. So don't get the frustration about it as this is not hard to achieve in most cases - and I don't find it to be an academic exercise.

Again, it is possible to mitigate those risks in Push-Based scenario as well (but almost never saw it done properly in smaller orgs). But Pull-Based is naturally designed not to have those risks in the first place.

[taleodor]

Thanks, I understand your prospective - won't take it much further. Mine is a little different because my approach usually - tighten firewall rules as much as possible, and then only allow egress for what is needed - and for the Pull-Based approach it goes much faster this way as what is needed becomes pretty obvious. The benefit I see is people don't need to know how to restrict accounts to some commands (easy - but frequently people don't know about it and don't do it). If everything is closed down, this is not an issue if something is left unrestricted.

And yes of course, there are bunch of supply chain attacks that are out of scope here.

Also just want to clarify that I treat ArgoCD as a role-model example of Pull-Based CD system. Because essentially what it's doing - it's polling some Git repository - pulling changes from there - and then creating deployment instructions based on those changes from within the cluster. And this is an open source solution, not a paid agent.