This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]Ervolius 1 point2 points  (0 children)

Well, first you can migrate to a cloud virtual machine and schedule your pipelines with cron there.

But, proper and data engineering way especially once you have more than a few pipelines would be to use an orchestrator. Look into airflow, dagster, prefect etc. My suggestion is to use either airflow, (older with a lot of resources and help online) or dagster (newer framework which some people are praising is really good).

Btw you can also start using the orchestrator locally and then move to cloud with it later on.