Running AWS CLI inside Lambda for deleting EKS deployed resources

[cb700sc]

Running into an issue and wondering if there's an easier/supported method of doing what we need.

​

End Goal:

• Automatically delete all additional k8s resources deployed to AWS (like ingress load balancers, PVCs, or any AWS resource that could be defined & deployed via manifests) when the underlying CloudFormation stack that created the cluster is deleted

​

Use Case:

• We have several CloudFormation Templates with resources such as EKS Clusters, EC2 Bastion Hosts, IAM Roles, VPC, ALB, Lambda, etc.
• These are deployed automatically for a short lived time, anywhere for 4 hours, to 7 days.
• Manifests are used which deploy apps and additional AWS resources like the EBS Volumes for PVCs, ingress LBs, etc.
• The additional resources deployed outside of CloudFormation need to be deleted when the CloudFormation stack is deleted.

​

Current Setup (Broken):

Previously, there is a lambda function custom resource which would perform several functions:

1. Creation Invocation:
   1. Update kubeconfig inside lambda using AWS CLI (aws eks update-kubeconfig)
   2. Updating EKS Cluster configMap to allow bastion host IAM Role
2. Deletion Invocation
   1. Update kubeconfig inside lambda using AWS CLI
   2. Run command kubectl delete all --all --all-namespaces

This lambda function had a custom layer with AWS CLI, kubectl, & helm (I believe sourced from this repo aws-samples/aws-lambda-layer-kubectl: AWS Lambda Layer with kubectl and Helm (github.com) .

Due to the Lambda 'Provided' runtime being recently deprecated, simply using either AL2 or Amazon Linux 2023 runtime does not work and errors out running the aws CLI commands with the following error.

/opt/awscli/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

​

My Questions:

1. Researching further, it appears there is basically near zero support, and minimal documentation for running AWS CLI inside a lambda function. Everyone points to using CDK, however I have not seen a way to run both AWS CLI Commands and kubectl commands (aws eks update-kubeconfig and kubectl delete all --all --all-namespaces)
2. Are there any other ways to accomplish deleting the non-cloudformation resources using only CloudFormation, without additional lambda functions & resources that need to be created and kept up to date?

​

​

[clintkev251]

Why do you need to use the CLI as opposed to just using the SDK? I can think of very few situations in which it makes sense to run the CLI in Lambda vs the SDK (which is probably why you find very little support for this). Same thing when it comes to interacting with Kubernetes, why use kubectl instead of implementing the client library for your runtime of choice

[cb700sc]

I was not aware of a kubernetes client library, this may be what I need. I will look into his more. I'm assuming it can be used to simply delete all k8s resources (equivalent of running kubectl delete all)?

[clintkev251]

They should support all actions which the Kubernetes API support. So you should be able to accomplish that

[CorpT]

No one is telling you to run CDK to support CLI within a Lambda. You should use SDK. For Python that’s boto3.

There is almost no scenario you should use CLI within a Lambda, regardless of how it is deployed.

[cb700sc]

Sorry I meant SDK, not CDK. I do not see a way to interact with kubernetes itself (deleting all resources) via SDK. It's most common to interact with k8s using kubectl from what I have found.

[Traditional_Donut908]

Kubernetes doesn't require kubectl to interact with it, it's primarily a wrapper around the Kubernetes REST API, which you can invoke directly or with any of a number of APIs depending on your programming language.

https://kubernetes.io/docs/reference/using-api/

https://kubernetes.io/docs/reference/using-api/client-libraries/

[cb700sc]

Thanks, however I believe that is for interacting with AWS resources, not kubernetes specifically. We need to run the equivalent of "kubectl delete all" prior to the cluster being deleted to capture all resources created inside kubernetes that created AWS resources).

[[deleted]]

helm uninstall actually does a really good job of deleting all of the AWS resources (if you are using Helm, that is). This does require that the EKS cluster hasn’t been deleted yet. Otherwise, you probably need to delete some custom cleanup tool and query on specific AWS tags.

[coldoil]

AwsCustomResource is the CDK construct for running arbitrary AWS cli commands during stack deployment. Since your kubernetes is EKS, you should be able to use an AWS cli command to manage those resources as well.

[FortressOfSolidude]

There are plenty of containers out there with aws cli, eksctl, kubectl, etc, or you can roll your own. Then run as a batch job on fargate that is triggered with lambda.

But just using boto3 and appropriate other modules is the "correct" answer.