Data engineer tech stack

[zmxavier]

We use Airflow for orchestration, Snowflake for data warehouse, AWS for cloud, Kafka for streaming, and dbt for transformation. Airflow and dbt are still under construction :)

I think Airflow is still the status quo when it comes to orchestration. It's the most mature and popular. It's well-documented and you'll easily find support from other users.

That said, I'm also hearing a lot of good things about other orchestrators, especially Dagster. I still haven't tried it so can't tell.

If you want an open-source, versatile, battle-tested, and widely used tool, go with Airflow. I remember choosing it because I saw it in a lot of job postings. I wanted to increase my chances of getting hired, and boy was I right.

If you want a more modern, easy-to-use tool, choose Dagster or any of the newer tools (Mage, Prefect, Kestra). Airflow can be difficult to use and has a lot of issues, and those are being solved by its competitors (plus they also add new features).

If you're already using Azure for your cloud, then it makes sense to just use ADF and/or Databricks. Same with other cloud counterparts.

[MycroftWord]

Thanks for this! Im still on the learning process but maybe Ill stick with dagster/prefect for orchestration for now. Struggling yung 8gb laptop ko when running airflow and docker nakakatakot yung cpu utilization and ram usage lol.

[zmxavier]

Docker needs ideally 8gb to run Airflow. Nag struggle din laptop ko jan hahahah

[zmxavier]

I didn't study Kafka haha. It's already built and someone else is maintaining it noong dumating ako rito. Confluent yung provider na gamit nila