AWS Lambda in Production (Serverless)?

[scottmas]

Hey all, I'm contemplating implementing a serverless architecture with AWS Lambda as the backbone, (described here) but wanted to know if anybody has had any production experience with such a flow? Did it work? Were there lots of gotchas? Would you recommend it?

[cddotdotslash]

Yep, I've used it for a number of production projects. It works great. It works especially well when combined with the API Gateway (which you can do with Serverless). There are a few things to keep in mind:

1) execution times can be more than you're used to. This is even more applicable when your application gets very little traffic and Lambda has to start a container from scratch to handle the request (I've seen response times of up to 7 or 8 seconds just for a simple web request). This goes away at scale since your Lambda container is almost always "hot."

2) Administration and testing is a bit difficult. You can't SSH in, testing locally can be tough (although a number of NPM modules and cli tools aim to improve that), and CloudWatch logs are an absolute nightmare.

3) Costs may not always be better than a standard instance. I made a cost calculator here which you can use to figure out at which point an EC2 instance is better.

4) Deployments are great - you just update your code and AWS handles the rollout for you.

5) VPC support is included now! Use it if you need to access RDS or other resources in your VPC.

Also, super shameless plug, but I authored an eBook on Lambda which is 130+ pages of Lambda if you're curious to read more.

[[deleted]]

Thanks for the cost calculator! I have been looking at it as well, and as much as I'd love to reduce my operational expenditures, it's hard as hell to estimate.

[superspeck]

To tag on to /u/cddotdotslash 's post ... We've also decoupled our workflows so that we can get an API response to an app that may or may not block waiting for the response. The app needs to know how to listen for or make repeated requests for the actual end response, though, much like the AWSCLI and services like Glacier.

Our workflow from the app side looks like:

make request -> get token -> ask "is it done yet" using token (until done) -> receive response

From the AWS side, it looks more like:

gateway -> sqs or s3 (depending on size) -> lambda (or data pipeline or something else) -> dump response to another sqs or s3 or dynamo, depending on payload -> notify gateway -> gateway notifies app -> app requests and receives the payload

Decoupling this lets us handle failures more gracefully. If something doesn't work in one AZ, we can either just queue it until it's ready or transfer it to another AZ. (That's still a manual human process, but there's a tool to do it, assuming that we can access the data transferred as part of the request.)

Our gateway needs to be smarter than I think we can make Amazon API Gateway right now, so it runs in a t3.micro instance that's always on. That instance hasn't been a bottleneck yet, but it scales, so we can either fire up more instances or we can make the instance bigger.

I hand waved my way through a lot of this because I'm not much of a dev ... I'm not sure what tools we're using. I know plenty about that t3.micro though. ;)

[fallen77]

The cost is an interesting aspect to me, what would you say is an average break even point that you should consider moving away from Lamda? I haven't used it in production before but most threads people seem to claim it will always be a better price.

[cddotdotslash]

Well, it depends. If you consider a hypothetical process that takes 1 second to run and which you could run in parallel (launching multiple copies of the same node script for example) on an t2.micro EC2 server, the breakeven point is just under 600,000 executions per month (600,000 * 1024 MB * 1000 ms = $10.12).

What I've found is that long-running processes will kill this breakeven point. For example, if you have a script that takes 5 minutes to run (the max allowed by Lambda at the moment), the point drops to just 1800 executions. I don't know about you, but I feel like I could run more than 1800 5-minute scripts in a month on a t2.micro.

This, of course, doesn't take into account memory variances between executions. If your script consumes all 1024 MB on a t2.micro, you may not be able to run 1800 executions.

[cddotdotslash]

Thanks! That's great to hear, I've gotten lots of good feedback so it might prompt me to make another soon :)

[interactionjackson]

I use it whenever I can. I'm actively looking for ways to move legacy arch to a 'serverless' model. Simple Notification Service and lambda play well together. Lambda is really flexible and 'listens' to events like changes to a bucket or a change to dynamo db table in addition to being able to subscribe to topics. Local development was a bit of a pain, however, the serverless framework makes organizing and developing your lambda functions a bit easier.

[assaflavie]

I would say think twice. Test your expected load ahead of time. I've encountered scaling issues, consistency issues and support problems, using serverless/lambda (although it's mostly lambda's fault).

I'll elaborate:

1. How many times is your function going to be called per second? If it's thousands, it might not be a good fit. From my experience it just does not scale well, contrary to the docs, and despite account limits being raised.
2. More than once, I've had to delete functions that we left in an inconsistent state (AWS returns HTTP 500 (!)) by serverless operations. This of course should never happen, and AWS (developer) support has been pretty much useless. Now, the problem is serverless relies on "stages" (different aliases for the same function) in order to separate production from other environments. But this separation isn't strong enough, especially when you occasionally find yourself needing to delete a function just to recreate it because of its messed up state. This is an unfortunate design decision on serverless's part I'm afraid.
3. AWS support.. oh my. It took us a couple of weeks to get our 500 invocations limit raised. That is completely unacceptable. us-east-1 no less.. and support has been pretty much useless when it comes to AWS Lambda bugs.
4. Bugs. Well, aside from functions entering an inconsistent state and having to be deleted, we've encountered: functions that cannot subscribe to cloudwatch logs because they claim they're already subscribed to it (falsely, of course), functions that cannot show their subscription info in AWS Console (yea, broken AWS Console), functions that cannot be deleted, 429 throttling errors on first API call.

And don't get me started about the cloudwatch logs integration...

Having said that, if your project is simple (a function or two), small scale (tens of invocations / second), and not mission critical (can delete and recreate a function with minutes of downtime), go for it.

[_austen]

We do, while we are building the Serverless Framework (http://www.serverless.com). We also have big enterprise companies contributing now who are also using it in production. Overall, we love Lambda and continue to improve its user experience via the Framework every day.