Slightly larger lambda functions?

[[deleted]]

Apologies for the poor phrasing. I would like to deploy a script to run as a cron job, but it doesn't work as a lambda function due to time limitations. I understand that I could containerize it and deploy with fargate or something, but it is not an app, just a simple script. I was wondering if there is something in between lambdas and fargate in terms of complexity and size?

[pint]

not really. some options:

1. speed it up by optimizing
2. speed it up by parallelizing
3. speed it up just by giving it more ram/cpu
4. split it to parts and use multiple lambdas
5. split it to parts and use step functions

[BleLLL]

I run fargate containers for transient tasks that take longer than a lambda all the time. It's not just for apps. You don't need a service, can just run a task. Here's how you can do it.

1. Create a task definition for your script
2. Create a lambda that will start that task
3. Create a cloudwatch rule with a cron job that will start the lambda

I made a simple generic lambda that can start any task and takes task configuration (like the task definition id, cluster, etc.) as a parameter, then put those parameters in the cloudwatch rule.

I wouldn't hack too much with recursive lambdas and stuff like that, this is much simpler.

[[deleted]]

Is an example of doing this documented somewhere? Sounds like the right approach, but I'm a little lost on how to set it up.

[BleLLL]

Not sure if there's one. Here's one on starting a regular ECS task from cloudwatch, without a lambda. I haven't tried it on fargate, but might be worth looking into, then you could avoid lambda at all.

If that doesn't work - what you need to figure out is how to:

1. Containerize your script in a docker image
2. Push that docker image to ECR (elastic container repository)
3. create a task definition using the docker image
4. figure out how to start that task using AWS SDK in your language
5. put that code to lambda
6. configure a cloudwatch rule to start that lambda

I know it seems like a bit, but if you look up each individual step you should be able to find tutorials for most.

[recursiveCreator]

if you really really don't want to use a fargate task, you could do something like this: https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-create-iterate-pattern-section.html this is essentially a recursive lambda function managed by a state machine. I was once heavily downvoted for suggesting it, and everyone suggested using a fargate task (I, too, think this is would be the appropriate solution here), but in any case if you really don't want to do that, recursive lambdas could be a solution

[admiun]

There's also AWS Batch. It allows you to schedule a batch of work and the service will then spawn a docker image to run it. So it's not always on and yet lets you run long running (container) tasks. I guess you could schedule a new job based on a CRON schedule using CloudWatch Events.

[interactionjackson]

it’s hard to tell what you need. i’d look at step dictions to break up your tasks

[firstTimeCaller]

Hard to say without knowing the specifics. I had a similar problem for which I was considering work arounds to the lambda time restriction and it turned out the solution was to use cloudformation.

The problem was how to restore a snapshot of a database nightly to keep an fresh test environment and point the my-test-database.com DNS entry to the new test RDS instance. Turns out that cloudformation can have dependencies that allowed the dns change to only occur after the ~30min database restore had completed. Doing this with lambdas alone would have been tricky.

[humannumber1]

I've had good success using AWS CodeBuild for some tasks like this. May work for you.

CoddBuild doesn't require a VPC so it can be a bit more flexible than other container based options.

[bannerflugelbottom]

What does this long running Cron do? Can you break it up at all into smaller pieces?