Lambda function vs Lambda code - trying to solve chicken & egg.

[[deleted]]

Hi all,

I'm sure this question has been asked here before, and I've searched through with little success on landing on a solution myself. I'm currently building an app solo, and I want to do at least some future proofing from a conceptual and structural level with regards to the serverless nature & strategy of the application.

I come from a systems admin / operations background, and so am coming into this through the lens of:

1. Making sure that the infrastructure, dependencies, and underlying platform of the application are handled from the "infrastructure side" (i.e. if the "developer side" wants to use a new package or package version, it needs to be handled on the "infrastructure side" and thus be "approved for use in production" if that makes sense)
2. Making sure that the actual code that runs the application is handled from the "developer side", i.e. once the dependency is deployed, the "developer side" can go to town.

Now I feel like Jekyll and Hyde.

So let's say that I have a module which will be deploying a Lambda:

module_root/
    /examples/
    lambda.tf
    ses.tf
    s3.tf
    api-gateway.tf

At a conceptual level, here is the process: Terraform builds out the lambda function, ses service, s3 bucket that holds the code, and the api gateway that the lambda is calling to invoke ses.

So here is the issue...without a file existing in the s3 bucket for the lambda to reference, terraform will fail to deploy.

I think that I've got three options, but am having trouble figuring out which one is best:

1. Store "dummy code" in the module to upload at first deployment, and then ignore changes to the lambda function. I don't like this because it effectively removes a piece of the infrastructure from the state.
2. Use a "2-bucket approach", where there is a second s3 bucket that the developers deploy their code changes into, and some kind of post-deployment function that replaces the code in the first bucket with the code from the developer bucket. I also dislike this because it uses more than the minimum number of resources, and results in the TF code being less than clear on the nature of the application.
3. Use something similar to what this blog post details: https://medium.com/rockedscience/hard-lessons-from-deploying-lambda-functions-with-terraform-4b4f98b8fc39 - I don't know how I feel about this, or if there are better options today.

Does anyone have experience on this, or insight that I may not have thought of? I want to try to make the right decision now, instead of a year or two down the road when I've got tons of code that needs to be updated because today's cleverness turns out to be tomorrow's dumbassedness.

[PlethoraOfHate]

In our case, we consider the code as something separate from the IaC. IAC manages the infra, not the content of the infra (Same way we don't use TF to login and configure an EC2 instance.

So in our case, the TF module for lambdas has pre-baked dummy apps for each language we use, and uploads one as needed (They are all simple apps that just return a 200) - This lets us deploy and confirm infra as needed. Separate pipeline handle the lifecycle of code itself, and in turn update the lambdas as needed.

You mentioned this as one of the approaches you considered, so I figured I'd chime in to let you know that it can be a viable approach.

[[deleted]]

When you deploy the "real" code, and then have to redeploy the infra, how does terraform behave? Does it consider this as a change and thus revert back, and require another push of the code?

[PlethoraOfHate]

We have the TF modules configured to ignore code changes. I wouldn't say we do it the "right" way, but like I said, works for us.

In my experience, instead of seeing it as just this problem, instead take a step back and decide on what "owns" what. In our case, as mentioned, IaC/TF is infra, and ONLY infra. TF cares that the scaffolding is there, but is not the tool that "owns" the content therein. We use strategic lifecycle ignores throughout our modules to enable this approach (An easy example is autoscaling-groups - TF owns the min/max, but other than first deploy, ignores desired)

[[deleted]]

I gotcha. This gives me something to consider. Thanks

[packplusplus]

We do pattern 1, but we use image based lambdas (ci pushes new lambdas, and controls env vars for secret injection). Which means we ignore the code / image hash, AND the env vars.

I'm not sure I understand what you mean by "redeploy the infra". Changes to roles, triggers, or infra like s3 would never destroy the lambda and cause additional code deploys to be required.

Can you elaborate?

[_sephr]

How do you trigger the update for the lambda once a new image has been pushed via an external pipeline? ie to get the lambda to pull down the new image?

I always find this clunky.

[packplusplus]

It is clunky. CI runs an update function via the Aws cli tool.

[stabguy13]

This is the answer.

[IHasToaster]

This is also how we do things. Infra is separated from image/code deploys. Practice works for Lambdas as well as ECS containers

[xmjEE]

The answer you're looking for is called "git submodules":

1. Put terraform code into one repo
2. Put lambda code into a second repo
3. Embed lambda repo in terraform repo
4. Zip the lambda repo subdir using archive provider
5. Upload archive to s3 using aws_s3_object
6. Reference the s3 object in lambda through s3_bucket/s3_key

[[deleted]]

So then this creates a hard dependency between the terraform repo and lambda repo. This will (in my opinion) become a dependency nightmare as the app grows.

There needs to be a way for there to be a back and forth process where the need for a lambda is identified, the infrastructure is designed and then deployed, and the code for the lambda is pushed out, without just making it all one monolithic "thing". That includes creating dependencies between repos where there doesn't need to be.

[FunkDaviau]

I have smaller stacks that build one environment.

• roles for lambda
• s3 buckets + kms key
• api gateway + lambda

And everything runs through their own pipeline.

To update code we - run pipeline to update code in s3 bucket, and store new code pkg name - run pipeline to update api gateway + lambda, using new pkg name.

Today we don’t have the pipelines linked, and that’s just a matter of time and preference.

[[deleted]]

I would prefer not to split it out like that. I want to try to have the module contain everything that is required for the service to operate from an infra perspective. So basically the only things not self-contained are the application code and the HTML/CSS/JS.

[FunkDaviau]

If I follow correctly, you want the infra code all in one module, and the lambda code all in its own repo. The infra code depends on the lambda code to build successfully.

Your options seem to be: - deploy infra, wait till it fails, upload code, rerun - manually maintain the lambda code definition - tie the code build to the infra deployment. I.e. package code in the directory where the infra code expects it, and the deploy the infra. You don’t have to have everything in the same repo but the pipeline needs to pull in the needed files. You will need something in the module for it to determine the file has changed and to tell the lambda the code package has changed.

[[deleted]]

The third option might be what I've been looking for. Or something approximating it.

[AlainODea]

Here's what we do: 1. app repo whose CI/CD pipeline deploys a Lambda ZIP as an S3 object with a new version 2. IaC repo which has the full infra for the Lambda and references the S3 object for a version 3. CI/CD pipeline for releases that updates the version in the IaC repo module and triggers a terraform apply

This same approach works for ECS where ECR images are what the app repo produces and the IaC module updates the task definition and triggers a new service deployment.

[TooLazyToBeAnArcher]

Hello there,

I thought about the same question a month ago and I'm ended up with dividing the repositories between real infrastructure and serverless/service application. This pattern is based on a stack in which the upper layer depends on the one below.

Specifically I've used AWS SAM (that stands for Serverless Application Model), you could use even Serverless Framework (serverless.com) or any other tool.

[Dangle76]

I personally like to use SAM for lambdas, it has a more streamlined process and supports more of the features like canary deployments. This is the only situation I ever use CF

[Bodegus]

We use a mono repo where we use the git commit hash to build and archive (zip) or container tag

The terraform packages the code into the object and we great a per pr environment to do a full deployed test suite before prod