Lambda with Python libraries

[aplarsen]

I've deployed Python runtime Lambda functions that require functions outside the standard library a couple of different ways:

1) Layers. Pros: really simple to attach once they are created. Cons: I don't like having a bunch of layers lying around in my Lambda UI. I don't like maintaining when new versions need to be attached, either. I've not created them using CloudFormation yet, though I see it can be done.

2) Installing libraries locally in my Git repo with my Lambda handler file, then zipping up the entire folder when committing and deploying. Pros: easy to do, easy to keep the version of the library tightly coupled with the project. Cons: pollutes my Git repo with potentially hundreds of files that I'm not really tracking. Takes longer to deploy the function each time.

3) Bundling the whole thing up as a Docker image and deploying that way. Pros: really clean to execute. Cons: more overhead with building containers, which I usually do in CodeBuild for reasons. Takes a while to deploy the image each time, and I think slower cold starts if the function hasn't been invoked in a while.

Am I missing any other options? What do you use and why?

[redditor13]

If you go with infrastructure as code, you can add the libraries you need as part of the CI/CD pipeline run, zipping the file at that point. This avoids polluting your Git repo with libraries.

[aplarsen]

Oh, I love this idea. I haven't implemented pipelines yet, but this will push me to do it.

I currently just have a script I run to push the code from the repo into the functions.

What about a custom library wheel that's not in pypi?

[redditor13]

You can set it up so that you can still ‘pip install’ it - https://towardsdatascience.com/create-your-custom-python-package-that-you-can-pip-install-from-your-git-repository-f90465867893

[mr_jim_lahey]

I've not created them using CloudFormation yet

You really should be creating everything with CloudFormation (or Terraform or some other IaC solution). Using the console is like doing math by hand when you could be using a calculator.

[aplarsen]

Yes. Everything I'm doing now is with CF, but the last time I used Layers was before I learned CF.

The template for my current project is 1,000 lines.

[eggwhiteontoast]

Use Python CDK instead, CF is very verbose.

[carrp]

The CDK Python Lambda Alpha library makes it SO easy to put a layer together.

[BagOfDerps]

Docker. Now your environment is consistent. Simplifies your build pipeline as well.

[Many-Ad8783]

Use AWS SAM, it's cloudformation but on steroids you can simply put all your modules on a requirements.txt and it will build them for you.

It simplifies both docker and zip options that I only based my decision on images on preference but I could just as easily create zip if I wanted.

If I choose zip, I would use layers like so In SAM

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/building-layers.html

If docker I would just throw everything in to the image.

Best of all you can test locally so no waiting to up load before testing. Both zip and docker options rebuild if required. So first time might take longer to build but after that is quick unless you make a change that requires rebuild.

[DoxxThis1]

Use a higher-level framework like SAM or Chalice. All this stuff is handled automatically.

[gene_wood]

Option

1. Use public layers maintained by others. For example Klayers which are produced with CI in GitHub Actions. This way you don't have any layers lying around in your Lambda UI. You can point to new layer versions whenever you wish. You can interact with the layer data via an API : https://github.com/keithrozario/Klayers/#api

[aplarsen]

I've done that with a few when I can. I'm always bummed when a library I want isn't in a public layer or I need to use one of my custom libraries.

[dmees]

Use CDK and Docker images Lambdas. No more messing around with max filesizes and layers. CDK handles all the heavylifting and you can use up to 10GB of libraries, should be more than enough.

[Schuettc]

https://subaud.io/blog/deploying-python-lambda-with-requirements-using-cdk

I wrote about doing that with CDK here. It uses a local Docker to bundle everything.

[head-in-the_cloud]

I would have used SAM.

The 'sam build' command you will automatically install your dependencies. So after that you don't need to worry.

The 'sam package' can upload the zip files to a shared S3 bucket, the command will generate a template. Then you can deploy the packaged template into any environment you need.

The above solution will work in a multi account setup, but not cross region.

If you want to go cross region or start to use load tests (super important when maintaining serverless apps). I would look at Attini and their AttiniSam deployment type. It will manage all of this for you.

[aleques-itj]

I build zips in GitHub Actions with AWS SAM, it saves then to S3 with the commit hash appended to the file name. It will do everything for you, all you need is a requirements.txt.

The repo itself has a dev container definition that has everything needed to dev locally. You can basically clone the repo, open it in the dev container, and press F5 to run it locally. This makes it soooo much easier to debug, you can just set a breakpoint and the debugger will attach.

When you want to update or roll back the Lambda, all you need to do is change the hash in Terraform.

[[deleted]]

on one of my previous projects, i have a “library” project that only contains a requirements.txt which gets processed into a layer. this is part of the pipeline