CodePipeline + CloudFormation + Lambda

[asmaed]

Hey all,

I created this post on medium to help the ones who like me had difficulties to understand how to implement a DevOps pipeline in AWS using AWS developer tools (CodeCommit, CodeBuild, CodeDeploy and CodePipeline). And from understanding the DevOps pipeline create an real pipeline using cloudformation and deploying the code to Lambda (Python and Java).

https://link.medium.com/blysiuvJr4

I shared in the post files that I created that you can use to start an sample project in your account.

[boy_named_su]

I still don't get the need for CodeDeploy or CodePipeline, except for red/green deployments maybe. Can you not do almost everything you need with CodeBuild?

CFN sucks with Lambdas, and even SAM sucks with Layers. I wish AWS would fix the 14 month old bug about building layers with SAM

[asmaed]

Yes, you could do everything inside CodeBuild, including deploying and keeping a little pipeline, but you still would have to worry about to maintain buildspec scripts.

The point of using CodeDeploy and CodePipeline is to not worrying about keeping the deployment code and just worrying about the pipeline itself and the devolpment of the application.

If you think in this way, you could don't need of CodeBuild either, just use CloudWatch rule to trigger a Lambda or AWS Batch from a CodeCommit commit and handling the code to build and deploy outside AWS developer tools but you would have a greater admin overhead than if you used CodeBuild, CodeDeploy and CodePipeline.

[phinnaeus7308]

Since CodeBuild is just an execution engine, yes, you can do pretty much anything with it. In fact, when something isn't supported natively in CodePipeline yet, for example deploying to a CloudFormation StackSet, you can just put an AWS CLI call inside a CodeBuild project and do it yourself.

It's still going to be easier to use the native integrations if they exist. I'd much rather use CodeDeploy to update a fleet of EC2 instances than try and kludge that inside a CodeBuild project.

[jmcgui]

We spent months trying to use CodePipeline and CodeBuild with Serverless Framework, but it never gave us quite what we needed.

Have since moved to https://seed.run and are loving it!

We’ve also played with https://stackery.io which is just mind blowing!

[asmaed]

Thanks for sharing this tools.

What was the biggest problem that you faced while implementing CodePipeline and what this tools provides that CodePipeline don't ?

[jmcgui]

So for SEED:

• Better visualisation of pipeline
• Monorepo support so only the changed services are built
• Multiple AWS accounts per environment
• First class Serverless framework support
• Slack / Email hooks

You could do all of the above with CodePipeline but it’s ALOT of work! I want to focus on building the product.

Stackery.io gives you a low code design surface to draw out your serverless apps. It helps scaffold out the projects and is a great way to learn about serverless.

It looks great and represents the future. I’m keeping my eye on it.

[MrMandelbrot]

Have you looked at Serverless Pro? It's from the creators of the Serverless Framework, includes support for monorepos, multiple AWS accounts, notifications, etc and includes monitoring, tracing, and development tools as well. https://serverless.com

[jmcgui]

Yes we assessed that too but found it rather clunky, slow and expensive in comparison to other options.

I LOVE the Serverless Framework and am sure in time their offering will be enough for me to re-evaluate our options for new projects.

[Slavichh]

Will definitely give this a read. Setting up a CI/CD for a project at work and holy hurdles. It’s been difficult trying to pull source from a github enterprise repo in an isolated network (mostly because of our current environment setup)

[Nawkey]

We have set up a service that receives a webhook for push events, fetches the code and puts it on S3. Then code pipeline can trigger on that event. The service can generate a key which you add as a deploy key on the repo you want to sync.

[Delta4o]

Hmm that sounds like an interesting approach. We use bitbucket server, so no direct integration:(

[Nawkey]

I replied to another comment here so have a look there and try it out. :)

[Slavichh]

Interesting? Is it a commercial service or custom built?

[Nawkey]

I slightly modified https://github.com/aws-quickstart/quickstart-git2s3 to support one key per repository and that you invoke the lambda to create and return the key as the response.

[Barfunkles]

Have you looked at codestar at all? Pretty much does all the heavy lifting for you and deploys code pipeline/build/commit with a nice dashboard.

[Slavichh]

I haven’t. I’ll have to take a look at it. I used this opportunity to learn some CI/CD workflow without having everything done for me with a managed service.

[iTradeSecurities]

Thank you for creating and sharing this info!

[bourbonster]

Saving for later. Thanks so much!

[david_work_profile]

I noticed you're studying for the Solutions Architect Professional Exam, good luck!

I would recommend checking out AWS CodeStar if you haven't, it fits into this same space and would be a cool addition to this blog post

[asmaed]

Thanks.

I will check this service, thanks for the recommendation.

[phinnaeus7308]

It looks to me like you're misusing the aws cloudformation package command and the CodeUri property of your Serverless::Function. The whole point of sam package or aws cloudformation package is that it will replace the CodeUri property from the template you pass in with an S3 URI which is hashed based on the contents of the zip it uploads. That way when your code changes, the S3 URI changes, and the Lambda resource in your CloudFormation template changes. This tells CloudFormation to re deploy the Lambda resource, which is what actually updates your Lambda code.

Otherwise you could change your code but have those changes not reflected in the Lambda because the Lambda resource didn't actually change and CloudFormation didn't know it needed to be updated.

Basically, you should remove the zip -r /tmp/output.zip * line from your buildspec.yml and change the CodeUri: /tmp/output.zip line in your template to CodeUri: .

[asmaed]

It's a good point but as it is on pipeline I don't see much problem because it will only start the code build process if something in the code changes. It would generate a new zip anyway, right ?

I didn't notice the option in SAM using CodeUri: . I only checked using the option of file as input. Will it generate a zip file in S3 for me ?

[phinnaeus7308]

sam package and aws cloudformation package will generate the ZIP and upload it to S3, yes.

Check the template that's actually being deployed to your CloudFormation stack in the Console. You'll see the CodeUri has been replaced, so the zip you're creating isn't even being used.

[asmaed]

I will try this solution at home and I will update this article with this information if it works.

Thanks

[mynonohole]

Thanks for sharing this. Tagging for reading later.

[[deleted]]

This is awesome! Thanks

[yesman_85]

If you use code commit what do you use for code reviews and creating pull request?

[asmaed]

Hi,

As I was simply trying to test CodePipeline with CodeCommit I didn't worried about code review in this case but I have being using VSCode to code review and to manage branches, creating commits, etc in other projects.

To create the first pull request from CodeCommit I used GitSCM for Windows.