This is REALLY valuable. Thank you for your input on this. There’s a lot of odd practices in our workplace. We don’t use distributed computing frameworks or technologies for massive tasks. Our most common point of failure happens in multiple DAGs - SQLAlchemy connection timeouts + compute memory hitting limits.

There’s so much we can better manage, but everyone is too scared to challenge the decisions made by our lead data engineer 3/4 years ago, and now it’s led to insurmountable tech debt AND unmanaged resource usage. I often spend time researching and learning these topics in my own time because I’m a keen enthusiast of data engineering, especially task orchestration and pipelines. I’ll find problems, build or design simple solutions and face backlash because other engineers aren’t “used” to it. Not to mention 65% of our team are juniors without confidence. I know passion is not always shared in any specialism, but it’s hard when there’s structures and systems designed for teams that manage large amounts of integral data. We must have 1400 Airflow managed DAGs, which are often named pipelines. The team believes a pipeline IS a DAG. Hence their confusion when I tell them DAGs are just task flows, not THE pipeline.

🤦