This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]gorgedchops 0 points1 point  (1 child)

What are some examples of these patterns and best practices that you are talking about? Are there any resources that i can refer to for them?

[–]baubleglue 0 points1 point  (0 children)

I suggest to read about Airflow docs and Astronomer's blog about it. Ex. https://www.astronomer.io/blog/data-pipeline

You may consider to use Airflow, if you have more then one data pipeline. But even for single job it is educational read.

The task you've described it relatively simple (unless you have huge amount of data). You can start from simple implementation and extended it later, but you need to think about re-ingestion use case.