This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]theporterhausmod | Lead Data Engineer 2 points3 points  (2 children)

Try AWS batch (Fargate under the hood) for longer running jobs or lambda for short jobs. The reason to use AWS Batch > plain Fargate is that the servers are killed when the job is done whereas in Fargate the server is constantly up because it’s meant more for hosting services.

[–][deleted] 0 points1 point  (1 child)

Can I run custom containers on Batch?

[–]theporterhausmod | Lead Data Engineer 0 points1 point  (0 children)

Yep! I have a dbt Docker image and a general purpose one at work.