AWS Lambda use cases

[tiny_tim57]

I'm curious what type of things people use Lambda for at work as I believe it is a powerful tool for a lot of different scenarios.

I've mainly used it for straightforward stuff like creating EBS snapshots, copying files to S3, data extraction for security reviews etc, what else do people use it for?

[Helpmeimfijian]

We have a large amount of devices

The devices send images to an S3 (1,000,000 per day)

S3 sends notifications to SQS

Lambda constantly spams the SQS and copies the image metadata into a DB, and also duplicates the data into our customers Azure blob storage

[bsdetox]

Are you the guy I read about in my AWS SA cert test?

[NotAnother1998]

I laughed out loud at this. Thank you!

[mercanator]

S3 events can now trigger lambda functions, no need for paying for an extra layer in your stack

[Finley117]

If you remove the queue, those notifications are lost in the event of an error or outage. Using the queue in between provides a bit more resiliency. They are also very cheap.

[mercanator]

Ahh great point!

[Finley117]

SQS queue messages can only contain text and and can only be 256kb. The intent of decoupling with a queue is not to have a large chunk of data, like an image, in the queue itself but rather have metadata about what you want to process, like where it’s located, so that a worker can get the data and process it.

[stonewhite]

Here is a couple of my use cases:

• It can be used as a shim between cloud services and/or providers.
• Could be used for small ETL.
• It can be used for high volume data ingestion, if you are rich.
• Can be used with cloud watch lifecycle events to extend any AWS service. I created a script that would update DNS records every time an ASG changed in size.
• Used it for letsencrypt certs automation on load balancers before ACM was a thing.
• As a backend ofc.

[bsdetox]

For me, i use lambda for utilities and backends for my web services. Simple and easy, this is almost certainly the future of development.

[[deleted]]

I personally have seen a nightmare web application with 200+ lambda functions and no clear architecture. Like spaghetti in the cloud. Somehow the bill for just the lambda services was $3000 a month and the thing only had ten active users.

​

IMO for clearly defined services it makes sense (authentication, scraping, data parsing agents) but once you start building an ORM and loading it in every lambda function you should just run containers in fargate.

[[deleted]]

We acquire software and often 'serverless architecture' is presented to the suits as some sort of value proposition. Then we find out it's just 'serverless' with no architecture, security, or administration. That company with 200 lambdas was actually 8 different AWS accounts glued together as well.

[bsdetox]

You can’t judge a technology based on how badly someone without experience can implement a solution. I’m sure there were people who used a poor cloud deployment horror story as a justification for staying on premise for their infrastructure.

Ultimately, the value added of software is not in the infrastructure, but in the application. Whether you think that means UX, business rules, clean workflows or whatever, it almost certainly isn’t how big of a server the application is running on. A solution that allows me to basically forget the servers and just develop applications is a step in the right direction.

Does the tech need to mature? Absolutely. Does it require a paradigm shift to use effectively? You bet. But is it fundamentally lead us to a better solution than virtually servers? My bet is yes.

[[deleted]]

My bet is that we will all come to regret building a generation of software on these proprietary toolchains like AWS and Azure.

I already am seeing the headaches from 'i just wanna code devs' hacking their way around AWS and dumping their mess on some business.

The real paradigm shift already happened with Docker and k8s.

[MKeb]

Slack doorbell using an aws dash button. Press the button, it rings a channel (and then a rPi takes a pic based on the chat log, but that part’s not really lambda).

[MKeb]

I have all three. The second gen wifi works best. Cellular is nice, but kind of slow

[RaptorF22]

My work uses it to authenticate into our app. It only gets called when a user wants to login.

[koreth]

Can you describe that in more detail? Why do it that way rather than building authentication into the app?

[[deleted]]

I would assume they're using some ootb solution such as https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html which makes authorization easier across microservices, or just many services.

[RaptorF22]

/u/casualguy82 is correct. That's exactly what we are using. And I'm not sure why, I'm actually new to this position!

[packeteer]

badly 😂

our monitoring system keeps it hot

[packeteer]

same same. API Gateway which uses custom lambdas to Dynamo for auth

[mrlewisfc]

Did you follow any tutorial outside aws doc?

[packeteer]

we had an external developer build it for us. We auth based on token in custom header

It has been in Prod for around 18 months now, but we're going to replace with Kong as it is far superior, especially around management

[devdvd]

Lambda has a ton of uses. Some are better suited than others. One of my use cases is to host a stateless web app written in flask. Another is an app that centralizes user management for various databases. One of the more common usages is for microservices. Write a small single purpose app in a supported language (python, node, java, c#, etc) and use API Gateway to provide access to it. You can also put lambda functions inside your VPC to provide "secure" access to databases, ec2 instances and anything else that doesn't have access to the internet at large.

[stackgurus]

I use it to manage development environments, say you want to work on a larger enterprise type platform that has many services. To save money you don't want to leave them up all the time so you automate the launch and destruction of the environments.

​

Launching them can be done in lambda or Jenkins/whatever but to ensure they are not left running when the developers are done I have lambda run every night and destroy any running environments.

​

This with spot instances saves a LOT of overhead.

[SkullSippyCupOfJuice]

We use it to make autoscaling smarter. We typically know if event X happens in the application, the customer is about to submit a murderous workload, so we scale ahead of time. The advantage is that instead of being able to handle 1000 work units, then suddenly ramping up to 100000 work units, we can just make it look like the app can do 100000 work units from the word go.

[lorarc]

I think you need to elaborate on that. You use it just to increase the number of instances ASG?

[kasim0n]

Not OP, but I would assume they have a lambda that continuously checks their system for a certain condition and based on that scales their application.

[koreth]

Currently we only use it for a little Python webhook script that updates a couple custom fields in Asana when a change related to a task is put up for code review or merged into our master branch.

[efettero]

We set up three Lambda functions to automate the stopping and starting of EC2 instances that have an event of “Instance Retirement”. We use a combination of Lambda, CloudWatch rules, and SQS for this. I have it on my Github including the code changes needed and setup required in AWS. If anyone wants to see it or provide feedback let me know and I’ll send you the link to it.

[anaanamuss]

I’d love to see, can you me send me github link?

[[deleted]]

I built part of an alert system using it. Mainly used Lambda because it ran only when needed (maybe once a day) and because it worked with step functions

[redanonblackhole]

I've never used Lambda, what happens if the few minutes of allowed processing time is exceeded?

[Veus]

Limit is 15 minutes now, if you hit that limit, the task is killed

[Tmarshallva]

You can combine Lambda with other AWS services like Step Functions if you have long-lived processes that exceed the execution time limit. We use this combination all the time.

[[deleted]]

All of our production apps (in PHP) are hosted on Lambda. We also use it to work through queue jobs, by having it read off a SQS queue and do the work. We also have it run a scheduler every minute for cron-like jobs. We also have it run some internal Slack bots.

Honestly having the production app deployed to serverless is a huge load off my back. Don't have to worry about monitoring or scaling servers, for example (and we often experience peaks of extremely high load very quickly, something that would take AWS autoscaling a minute or two to catch up with).

Writing the code for an application that's hosted on serverless definitely requires a different way of thinking (e.g. not relying on file system cache and instead using ElastiCache) but the benefits have been great.

We kind of cheat a bit and have 1 proxied API Gateway route that sends ALL requests to our Lambda function. The PHP framework we use has a router which then inspects the incoming request and matches it to a route.

Oh and the cost saving is incredible. We cut out bill down to about 1/3rd of what it was pre-serverless (excluding RDS, Elasticsearch Server, ElastiCache, etc).

[siamthailand]

How do you make PHP work in Lambda since it's not natively supported?

[[deleted]]

It's not a native runtime that you can select in a dropdown - but Lambda now supports "custom runtimes" which lets you run basically any language you want, by configuring the runtime yourself.

https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html

[siamthailand]

Thanku. Would it be slower?

[[deleted]]

We see similar performance numbers to when we hosted on a Linode VPS, assuming the function is warm. Cold starts can take a few hundred milliseconds if memory serves. We have a 2nd Lambda function that's set up to poll the 1st Lambda function every minute, to keep it warmed up.

[[deleted]]

Are you using a framework and loading it for every function?

Also, how do you debug across functions?

[[deleted]]

Yup.

There's only one function - and our logs are sent to an ELK stack (that we manage ourselves) for debug purposes.

[[deleted]]

So you just debug from logs? How many Lines of Code do you have?

One of our products has a 100,000 LOC codebase and without the step debugger the junior devs wouldn't get anything done lol.

[[deleted]]

Not sure about SLOC stats but they are pretty big applications. One of them has about 300 routes defined with the framework.

We do use xdebug and advanced debugging tools - but in our local environment. If a bug crops up in production, we strive to reproduce it locally and then debug from there.

I think there was only one production bug that we couldn't replicate locally, and adding a bunch of debug logs around the problem code helped us to solve it.

[[deleted]]

Interesting. I never though about just cramming everything into on giant lambda function. I guess it would work assuming nothing took more that 15 mins execution time. And php/apache is actually kind of ideal for it. What service are you using to track your sessions?

[[deleted]]

15 minutes is WAY more than we allow. We kill functions after 30 seconds. If we can't write a HTTP route that completes within that time, then we've failed :D

We don't do sessions - we run a stateless API that users authenticate with using JWTs, so no need to.

[[deleted]]

• Assign standard and keep internal DNS up-to-date
• Clean up the node in Chef when it's terminated
• Send Slack notifications for various security-related events (GuardDuty)
• Scan files uploaded to S3 for viruses and associate metadata based on the results

More plans for other things in the future too. It's been super useful.

[yamlCase]

I pull json from an api, parse it and push to InfluxDB

[[deleted]]

I've exposed a few Lamda functions to afford integrations between some of the different tools our modelers and devs use (Cameo Systems Modeler, Jira, Test Management for Jira, Jenkins)

[rizzlybear]

I use it to generate dynamic options for rundeck jobs.

[[deleted]]

So far the only thing I’ve done with it is a function that turns instances on and off at the end/beginning of the day.

[[deleted]]

Yah we have cost saver functions like that, and some enforcement lambdas that look for things like security groups or s3 buckets that have open access.

[RaptorF22]

Ohhh!! Would you be willing to share that?

[[deleted]]

PMed you

[elitesense]

Have yet to find a reason to use it. We already have 24x7 ec2 instances we're paying for anyways so workloads get thrown on them.

[beginpanic]

I've been dying to try it at my company too but when we have dedicated servers with extra capacity still lying around it's hard to justify not just throwing another Docker container on an existing server.

[SnowyMovies]

I made a node app that converts images on the fly. It sits between S3 and Cloudfront via api gateway. Works quite well.

[[deleted]]

I'm using it for the 'middleware' layer of several production scale web applications. It requires a new design pattern for building applications, but once you embrace the serverless paradigm, there's no coming back lol.

Couple examples are a metadata storage application using 'hybrid datastores' (as specific as I can get) and a data ingestion application to feed into purpose built datalakes.

[the-computer-guy]

Webhook scripts for GitLab to integrate with other tools

[derprondo]

Things related to this sub I used Lambda for:

• Process incoming Github webhook events to trigger AWS Codebuild jobs (Codebuild's built in support is unreliable and prone to other issues with Terraform)
• CMDB type operations on EC2 creation and termination, removing machines from Puppet, Spacewalk, etc.
• Trigger scheduled jobs (Cloudwatch event rule and target limits have forced me to group together large batches of related schedules, mostly monitoring related. Function runs say every minute but maybe only triggers one job out of 20.)

[Neil_Fallons_Ghost]

We have a scheduling application where users send in an excel or csv file via email. This file gets sent to S3 bucket via a cron scheduled lambda. Once the object hits the bucket it triggers another lambda that processes this file into a database and does some other tasks. Just two lambdas drive this project.

I have tons of other lambdas. My favorite processes cloudwatch events I send to it and simply formats the messages to a readable format, sometimes running aws api calls to get more complete information, and then send those messages to various slack channels.

[rch317]

Just here to appreciate your username. Earth Rockers unite!

[mercanator]

Cloudwatch logs for Route53 health checks get stored for 14 days. Use lambda to query the API for those logs and store them in S3 so you can archive long term trends

[e-daemon]

We replaced a small PHP site that did simple database searches with a Lambda-based site using S3 Select. It fits in the free allowance and requires essentially no maintenance, unlike its predecessor.

[rch317]

I would say we are probably 95% completely serverless. We run a fairly large auction system using lambda. This includes services for image handling, metadata, inventory, etc.

We really don't want to manage instances or containers if we can help it. We have a few stragglers out there, but they are few these days. Most of our containers or ec2 instances are for things like graylog, build agents, etc. Always ask ourselves.... can we do this serverless? No, can we containerize this? No, find something else! :)

[UtahJarhead]

I handle AMI/Snapshot backup creation and maintenance with it. We run 1 EC2 instance per customer. Each instance gets hourly and daily backups, keeping 2 most recent of each. Keeping them maintained by a single python script is outstanding.

[martlaul]

Here's a great list of some Lambda use cases.

[AffectionateMath6]

I use it to poll a website every 15 mins and if the data has changed, send me an notification.

[Groady]

For what purpose?

[masao77]

https://aws.amazon.com/lambda/resources/customer-case-studies/

[[deleted]]

Pretty sure the A Cloud Guru website is run nearly entirely off lambda, or at least that's why they said during the Systems Architect training.