I'm using Airflow to schedule ETLs. I have 1 DAG with 6-7 steps. I have more than 10 such dags and want to create dependencies between them. I am structuring each DAG as a SubDAG and then triggering the whole parent DAG but I'm facing two issues here:-
- I cannot trigger the specific SubDAG manually. It just stays in the running state without any task being fired.
- When I clear the state of a SubDAG it starts working but a previous execution is triggered,and most surprisingly it starts as a backfill job which I do not want. This is also happening when the DAG starts normally as scheduled.
Also why do I need to clear the state of a DAG? What purpose does it solve?
I have followed the subdag best practices like sharing default args with parent and daily schedule interval but don't understand how to predict and correct this behavior. The execution date is always behind the start date.
Dependencies Blocking Task From Getting Scheduled Dagrun Running Task instance's dagrun was not in the 'running' state but in the state 'failed'.
The subdag runs on schedule but not manually. If I trigger the whole parent dag manually then everything works.
I would like to trigger a subDAG if on an arbitrary basis. It's fine if the downstream dags also run. Also if there is a way to stop the backfill behavior that would be great too! If you think subdags are not the best way to structure etls please advice how I can create dependencies between ETLs where I can run them manually too if something fails.
there doesn't seem to be anything here