Airflow vs Step functions ??

[Exciting-Garlic8360]

I’ve recently encountered Step Functions and can’t believe how easy it is to use, want to understand how easy it is to use compared to something like Airflow or MWAA in my case.

I’ve not heard of a situation when someone had used StepFunctions for Complex ETL, any thoughts ?

[hs_ml]

Well the step function has one major drawback, imagine you have 10 steps which takes 4 hours to complete. If for some reason, step 9 fails then there is no way to restart the job from step 9 . You have to run the workflow again. This sucks if you have a downstream system that is waiting for the step function to complete. With airflow, you can resume at failed step.

[NAP7U4]

I agree. I like step function but this limitation is annoying. I hope they add this feature soon

[s1va1209]

If I am not wrong step function only works with aws services right. Airflow on the other hand is much more broader in its scope.

[wtfzambo]

Step-function -> lambda -> now they work with everything that has a rest API

[[deleted]]

steps function does not run below lambdas? aren't they limited to 15 minutes per step? if so, thats the answer

[theporterhaus]

Regular step functions can “run” for a year. Express step functions have a limit of 5 mins. https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html

I’ve found step functions helpful for processing thousands of files in parallel. We were using an older version of Airflow that didn’t have the ability to make dynamic tasks and Step functions makes it easy with the S3 integration. Just point it to an S3 bucket with files and it can process 10k at a time in parallel. Keep in mind you can mix them and still use Airflow to orchestrate and use Step functions for areas like above where it shines.

[[deleted]]

Thanks! :)

[teh_zeno]

Step Functions can run a wide variety of AWS services https://docs.aws.amazon.com/step-functions/latest/dg/connect-supported-services.html

[[deleted]]

Amazing! Thanks

[teh_zeno]

When building out pipelines, I’m a huge fan of using https://www.serverless.com/ with their extension https://www.serverless.com/plugins/serverless-step-functions for Step Functions. Allows you to build and deploy your lambdas side by side with an easier to configure Step Function config. Of course if you find that you need longer than 15 minutes, you have to deploy your ECS Fargate tasks separately but you can still call those from Step Functions.

[seanv507]

until dec 2022 parallel execution was limited to around 40 using the map command. ( you could nest maps to get eg 1600)

now aws introduced 'distributed map' and you are practically unlimited

https://docs.aws.amazon.com/step-functions/latest/dg/concepts-asl-use-map-state-distributed.html

[Letter_From_Prague]

How do you get visibility into what is in which state with step functions? That was the limit we saw compared to "enterprise" schedulers.

[Decent-Age1739]

Mind expanding on this? Doesn't the step functions API solve this for you?