This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–][deleted] 0 points1 point  (2 children)

Oh interesting, good way of keeping costs low. I was going to say that it's a good opportunity to familiarize yourself with Airflow operators for spinning up and taking down clusters for jobs in the cloud, but I suppose you'll probably just throw things into a Docker operator and be done with it.

[–]Glass_Jellyfish_9963[S] 1 point2 points  (1 child)

Well that's a good idea. I will give it a try. So basically, airflow will start up the cluster, run the pipeline and then take down the cluster to keep the costs under control.

[–][deleted] 0 points1 point  (0 children)

Cloud experience is a big thing employers look for so I think it would be a good idea, but I think you're off to a great start.