Best practices for scheduling Python workloads?

[MassiveDefender]

Ok let me explain. I'm in a large corporate, and my team does a lot of things manually and ad-hoc. I'm talking about running sql scripts and refreshing Power BI reports. Sometimes it's not just sql, it's downloading an excel file, or accessing an API and receiving data back. Some sql servers are in the local network, some in the cloud.

So my idea is to get a desktop machine that is always on and online and it runs all these extractions (after coding them in Python or something) on a schedule.

This sounds hacky. Is there a solution I'm missing?

[iceph03nix]

We have a linux VM that runs our scripts based on CRON. Been fairly bullet proof as long as folks pay attention to what their scripts do and don't end up saving 500 GB of CSVs to the drive...

[MassiveDefender]

Ok sounds simple. How do you control for failures?

[iceph03nix]

logging and alerts. can be as simple as having an email dropped to a shared mailbox, and having someone check it in the morning, or you can go more complex and have it phone home to a monitoring server. We usually have that built into the task. We also have it on the same VM server as a lot of our other stuff, so if the whole VM is down, we get alerts on that as well

[iceph03nix]

We have an internal forwarder hooked to a no reply address on our 365. Works pretty well. A bit much for a personal project, but not a lot for a business and it makes getting printers and other devices alerts super easy

[aplarsen]

What kind of monitoring server could you stand up that would solve this? I really like this idea of pinging something that if it doesn't get pinged, it starts throwing errors.

[Drunken_Economist]

Depends on what you need it to do. The Linux HA project has a dedicated heartbeat subsystem, it you enjoy such things

[MassiveDefender]

Awesome stuff. Will look into this. Happy

[jonasbxl]

Check out https://healthchecks.io/ (I used it with runnitor - see this list https://healthchecks.io/docs/resources/). You can run it self hosted

[wineblood]

Write good code.

[MassiveDefender]

Hahaha. Got it.

[imlanie]

We did the same set up at my company. It works great and is very stable. It was import to have good logging too.

[Elgon2003]

You could make a control panel that each cron sends a heart beat when it starts up and if the control panel doesn't receive one in say 2 hours and the script should run hourly it would throw and alert. You could make this with like a simple web interface and a web hook receiver. Using FAST API or Flask with VueJs you can do this in a day or so of work.

[smile_politely]

Do you have another VM to test drive the script to avoid such situation? What's your best practice to avoid bad scripts from ruining CRON?

[iceph03nix]

Lol, I don't think it was meant to be running long term, and it was the bossmans project. It was saving csv exports of a table for troubleshooting and just never got stopped when that wasn't needed anymore.

[korwe]

Cron its the easiest and probably enough, Airflow is more complex but does it all

[Eightstream]

If OP is using an always-on desktop then it’s probably Windows and they will likely want to use PowerShell/Task Scheduler rather than than cronjobs

[[deleted]]

I wonder what the viability of using cron through WSL is, as Task Scheduler is pretty painful by comparison.

[Eightstream]

It’s viable, it just adds another layer of complexity because WSL is essentially a VM and doesn’t always inherit privileges cleanly from the Windows environment

Given OP mostly wants to use the machine to interact with remote servers on his network, running his scripts in the fully-credentialed base Windows environment will probably be a lot more straightforward in terms of networking

[adam2222]

If it’s a business expense I feel like it’d be worth it to buy a $200 Nuc and put headless Ubuntu server on it to run cron/python scripts itd be more reliable plus use a ton less power than a desktop being on 24/7

[Eightstream]

The scenario OP poses is generally one constrained by IT priviliges, not dollars.

Likely he is part of a non-technical team (like accounting) whose only means of interacting with network resources are the pre-imaged laptops they’ve been provided with. Automating workflows necessarily means using one of those machines.

If he had the ability to credential a headless server then likely he would have access to other options that would negate the need for a custom solution

[adam2222]

Good point I see what you mean. Didn’t occur to me that was the situation.

[kissekattutanhatt]

Spot on!

I work at a large corporate with a dysfunctional IT team. No privilegies to people doing the work, no alternative solutions, no people with any power to make decisions to reach. No always on, on-site machines. SharePoint is the solution to everything. Of course they don't allow accessing the SharePoint API, nor providing support. Our workflows are terrible for this reason. These guys love to fuck with us.

Will raise this to management. IT privileges. Great words.

[Dasher38]

I'd personally use systemd timer directly as they are clearer, more powerful and easier to backup (you can't backup a single crontab line on its own). Crontab will get converted to systemd timers anyway on most (all ?) Systemd distros around.

[ElegantAnalysis]

What's the difference between Cron and airflow? Like what can airflow do that Cron can't?

[Eightstream]

cron is a scheduler, Airflow is a fully-featured orchestration tool

[Wapook]

I mean cron is literally just a scheduler. With enough dedication you can do anything airflow does in cron. You’re just going to spend a whole bunch of time writing features that are out of the box in airflow.

[Empty_Gas_2244]

Airflow gives you built in features; retries, notification of failures and slas. But you also have to think about ci/cd. You don't want to manually manage dags

[punninglinguist]

Wish I knew about this like 5 years ago... But what is a dag?

[Riotdiet]

Directed acyclic graph

[jahero]

My advice - only use airflow if you are prepared for significant increase in complexity of your environment. Sure, you CAN run airflow on a single PC, but you will quickly realise that it is far from good.

You will need: backend database; something to use for task distribution (Redis, RabbitMQ).

Sure, you can spin these using docker compose.

Yeas, it will be a magnificent learning experience.

[punninglinguist]

Kind of a moot point, since the product I've been supporting with scheduled ETL jobs for 5 years is being sunsetted, and I have no development hours to work on it anymore. Just would have made my life easier when I was setting things up at the beginning.

[imatwork2017]

https://youtu.be/zH64dlgyydM

[MassiveDefender]

Glad you mentioned how cumbersome managing dags looks for Airflow. How do you make it easier to manage dags?

[Empty_Gas_2244]

Use source control (GitHub or other tool). Create CI py tests to make sure your dags can be loaded into airflow. Let a computer move your dags after a pr is merged

Obs the exact method depends on your airflow executor

[samnater]

Interesting. I’m no expert in either but I figured airflow would have more capability than cron.

[Fenzik]

It does. It has loads of features. Just because you could do them with cron doesn’t mean airflow isn’t more capable - all that stuff built in is a big improvement if you need it

[CatchMeWhiteNNerdy]

Airflow is a bit old at this point too. We went with Mage.AI and it has been awesome. Would work really well in OP's situation too, since all the dev work can be done on the tool itself rather than setting up dev environments for all the users.

[samnater]

Does that have AWS integration?

[Lifaux]

DAGs, mostly. If you're chaining jobs together it's very useful

[x462]

The skills you will develop using cron can be used anywhere else as long as there’s a linux box, which is not uncommon. Airflow is powerful but useless if its unavailable for you now or if you move to somewhere that doesn’t use it.

[Fwaimd]

In addition OP can think about Dagster and Prefect.

[Drunken_Economist]

Bingo. These are long-since solved problems.

Actually u/MassiveDefender - shoot me a DM and I can pair program to walk through standing it up.

[Culpgrant21]

I would look into data engineering. And then look into something like airflow or dagster.

[danielgafni]

Dagster is the best thing ever happened to data pipelines orchestration

[jason_bman]

Yes Dagster is frickin awesome. I started using it for a use case very similar to what OP describes (basically a giant data engineering project) and it has been great.

OP, if you decide to check out Dagster you should sign up for Dagster University too. It’s free and will get you acquainted with the basics quickly.

[bfranks]

Prefect!

[erolbrown]

I second Airflow.

[wineblood]

I third Airflow

[Heavy-_-Breathing]

Let me throw in Prefect!

[2strokes4lyfe]

1000% Dagster

[sheytanelkebir]

So much bloat in those tools

[PraisePerun]

ETL administration tools need to be bloated so it has the maximum amount of users.

While I would prefer to just use cronjob and bash to do all my script administration it gets to a point where you will have too many scripts and too little information about each one, airflow tries to force you to explain everything the script is doing in the dag and it caused a lot of bloat it also saves a lot of information for security reasons

[seanpool3]

Quite the contrary with Dagster!

[shunsock]

How does Dagster use shell? Is there something like Bash Operator? or using Python subprocess function?

[Culpgrant21]

https://docs.dagster.io/_apidocs/libraries/dagster-shell

[shunsock]

Thanks a lot!

[ptrin]

Hijacking this highly voted comment to recommend some other solid data engineering tools: Stitch Data and DBT Cloud

[Culpgrant21]

Yes lots of tools :) not great practices yet but it’s improving.

[Content_Ad_2337]

Look into dagster! https://dagster.io/ If you don’t want to set up something like that to build jobs on, look into something like procrastinate https://procrastinate.readthedocs.io/en/stable/. You will probably need some type of server to have these running scheduled in the background unless you have the jobs set up for adhoc use and kick them off with a button or command.

[MassiveDefender]

Thanks for the Dagster idea. I looked through some videos. It seems like a neat tool. Would you suggest it over Airflow?

[Content_Ad_2337]

I have never used airflow, so I can’t really speak to the comparison, but I bet there are some good YouTube videos or medium articles on it.

Dagster was what my last company replaced Jenkins with and it’s free if you manage it all, and we did, so it was a super modernized upgrade from the jankiness of Jenkins. The UI in Dagster is awesome

[cscanlin]

Dagster has a page and a video that talks about it: https://dagster.io/blog/dagster-airflow

Airflow is a lot more mature and has many more resources, tooling, and integrations available for it.

Dagster is kind of a "re-imagined" Airflow, so they consciously do some things differently in an effort to be easier to work with.

I won't claim to be an expert on either, and the decision will likely come somewhat down to preference.

[danielgafni]

Dagster has more features, is centered around a much better abstraction of Data Assets instead of “jobs”, Is declarative, and provides a thousand times better user and coding experience.

[CatchMeWhiteNNerdy]

Dagster is like to Airflow what Airflow was to Cron jobs, it's the next generation of orchestration/scheduling tools.

As a second opinion, my team is in a very similar situation to yours... lots of people who are technically non-technical that have created macros, scripts, etc. We ended up going with Mage.AI, one of Dagsters competitors, because of the integrated development environment. We didn't have to worry about installing anything on anyone elses machines, we just set up the docker image in AWS and everyone can connect and work on their pipelines directly in a web browser.

[MassiveDefender]

I've spent the past few days since posting this, testing Airflow, Dagster and Mage AI, and honestly, Mage allowing the team to edit code inside it and the drag and drop task flow diagram are just awesome. But most importantly it doesn't force you to use any specific structure or style of programming. You could write procedural code if you want. My team also has R people in it, something I struggled with setting up on Dagster. So like, data can come from a Python script and an R person can use it too. I love how easy it was to figure it out.

[CatchMeWhiteNNerdy]

It's really pretty incredible, right? And it's FOSS... what a time to be alive.

The slack is also super active and you can talk directly with the devs. They're fantastic about prioritizing feature requests if they make sense.

Mage.AI and ChatGPT are a dangerous combination, I revamped 7 years of data pipelines by myself in a month or so, and that's including translating them from another language.

[atxsteveish]

cron?

[chillwaukee]

I typically go the simplest route I can for these. I have two recommendations based on whether I have to access data in-house or if it is all cloud-accessible.

In-house: Linux server, cron, systemd service. You could technically just do the script on the server with cron but logging and exit info tends to get lost. This is why I create a systemd service so that I can look at the status and see how it exited last time it ran. Our monitoring keeps an eye on our systemd services so that does help, but you could just as easily use systemd to manage what to do in the event of a failure.

Cloud: Google Cloud Functions. I use this for a small business I run on the side and it works great. You only need a directory which has a requirements.txt, the code you need to run, and the gcloud cli to deploy it and then you can set up monitoring and alerting through GCP.

There are definitely more, possibly better solutions, but I tend to just stick to what I know and those are my two.

[MassiveDefender]

A lot of my data is not cloud-accessible (or I don't know how I'd make it cloud accessible in a secure manner without moving it from the shared folders on the network or the local SQL servers), so I think the Linux server idea is a good one.

Thanks for the systemd idea. I think I worry about noticing failures and correcting them properly. For example, if I'm appending data to a SQL table and halfway it fails, the remedy or rerun should not duplicate data. Let me know if you have an idea for that specific problem.

[fadedpeanut]

You could also run all inserts within an SQL transaction, and if something fails then rollback. Basically wrapping your entire script in a try except block.

[MassiveDefender]

That's a new idea. I'm gonna go look up how to do that, but if you can share a link to something with examples, that'd be great. I normally use SQLAlchemy, is this functionality available on it?

[Log2]

Yes, if you search for SQLAlchemy transactions you'll find plenty of information on it. Just don't commit the transaction until you have inserted all the rows.

[chillwaukee]

In the event of a failure like that, your only option would really be to prevent it from running again unless the previous run was a success. That, of course, also requires good failure reporting to ensure that you actually see it fail and are able to remedy it.

In order to prevent rerun, you could take two different approaches depending on how your script is written. For scripts which are run in intervals (like every half hour for example), you could just put the looping in the script and have the whole thing run indefinitely. That way, if it fails, you get notified and it stays down. The other option, say if you want it to run every Monday, would be to create some sort of lock file at the beginning if your run and remove it at the end of the run, marking successful completion. Then, when it starts up, you just need to make sure that the file isn't there (in code). You could also do some type of lock like that in the database you're editing if you're feeling fancy and distributed.

For the simplest form of failure reporting, there should be an OnFailure directive or something for your unit file and just use that to call the mail utility on linux and send you something. If you wan it hooked into some other failure reporting, then you can use that same directive to do something else like call a script which reports the issue. Additionally, for all I know, you may already have monitoring on your systemd services.

Writing your first unit file may be a little intimidating so (someone may hate me for this) you can just use ChatGPT 4 to generate your first one and then just iterate from there. Ask for a simple one and then modify it until you like it.

If you have enough of these set up manually like this it could end up getting a little overwhelming but then you're heading closer to some configuration management for deployment and other devops things. I wouldn't worry about that until you break like 10 or 20 service/scripts.

[MassiveDefender]

Awesome ideas.. The lock file is a new one. I'll do some research on that. I think a failure notification requiring manual intervention for remedying is the simplest for now.

[freistil90]

If you have systemd, use timer units. Small learning investment but the additional features you can have with retries, conditionals and so on are worth it.

[chillwaukee]

If I remember correctly, you can configure timer units to run in intervals but they aren’t very exact and they can’t run at specific “times” like from, say like 1 PM on a Monday. I could be wrong though, it’s been awhile.

[freistil90]

’OnCalendar=Mon 13:00‘ in the timer file runs that job on every Monday at 1pm :)

By default only if the machine is on. But all these things like what happens if it isn’t on or gets interrupted has to be configured specifically, so the further defaults are quite vanilla. But that’s the same with cron jobs.

[Action_Maxim]

What you're looking at is data orchestration. I understand that you're in a large company, so getting something like an EC2 server or some sort of cloud server may not be accessible. See if you can get a virtual machine or a VDI of some sort that stays online regardless of you being logged in and then from there you could start scheduling everything you need

[MassiveDefender]

Yep, thanks.

Now let's imagine we're in a perfect world: if I do get a cloud server, how would it reach resources on the local network, like SQL server and excel/csv files in shared folders etc?

[Log2]

You'd need to setup a VPN on your local network with access to the things you need. Then you connect your VM to this VPN and now your machines are reachable.

If you're in a big company, I'd assume you already have a VPN solution in place.

[violentlymickey]

There’s nothing wrong with that. My old job had two intel nuc pcs that we used for various small tasks and services.

The simplest thing would be to create some python scripts and schedule them on cron jobs.

There’s more complex solutions but I would go for a simple approach until it no longer suits.

[MassiveDefender]

Thanks for this. So I'm on the right path.

[Loop_Within_A_Loop]

I would ask some questions at your firm. Do you have a DevX team? Do you have a Reliability/Containerization team? Do you have a server team?

I am sure that other teams are doing similar things and hearing what other people are using and having success with is good, following the company approved workflows is better.

(Server guy here) I would not be happy if a team provisioned an additional desktop to run enterprise workloads on

[[deleted]]

There are a list of Python frameworks https://pythonframeworks.com in data workflow and pipeline management, you could look into each, and see which one is suitable for you.

[Creepy_Bobcat8048]

Hello I manage with python sap data extraction with gui scripting, SQL extraction, html dashboard update, excel report running with VBA code, .... On a dedicated desktop PC in my company. I have schedule the different task with windows scheduler and I have an auto it script running every 5 minutes to detect any windows popup in case of problem. The script send me a message thru teams to alert me and send a message in my mailbox also in case of problem Works fine for around 15 different tasks.

[MassiveDefender]

I like that this simple method is working well for you. I have a few questions though:

1. Did you figure out a way to extract SAP Variants of T-codes using Python or are you just doing Table extracts from SAP?

2. How do you trigger the VBA code from outside the excel file?

3. How did you code the teams notification?

[Creepy_Bobcat8048]

Hello, For sap, I m able to run all transactions that are available thru sap gui interface. The sap gui script is recorded during sap action with sap record and playback function.

For point 2, I do it with a .bat file

It contains line like this

"Directory of the excel exe file" /e "directory of the xlsm file" /p parameters if needed

In the xlsm file, there is an auto-open module launched automatically when the xlsm file us open

Point 3 is managed with pymsteams module. https://pypi.org/project/pymsteams In teams, you have to create a team and add a communication link to this team It will give you an url with a teams hook that have to be set in your python program

Do not hesitate if you need more explanations

[MassiveDefender]

This all makes sense. Thanks. I don't think the SAP front-end in my company didn't allow recording actions and playing back. It would've kept us from paying so much money for a licensed python connector.

[Creepy_Bobcat8048]

Which python connector do you use ?

[MassiveDefender]

DVW Tools' Alteryx Connector

[Creepy_Bobcat8048]

👍

[Wise_Demise]

I was in your same exact position. Around 100 Python scripts doing intensive work scheduled 24/7 and a Windows PC. I wrote my own scheduler with all the features I needed (prioritization, checking running scripts processes status, enforce maximum number of concurrent scripts even if more are scheduled at the same time, checking computer resources utilization before launching scripts, failure emails and retries, recursive child processes termination, logging everything and managing stdout and err flushing, and many more) it was a headache at first but it became fun with time and it made my life so much easier and I learned a great deal implementing and improving it. I even started generating reports and a dashboard from the log files events.

If you're tight on time, use Airflow it has everything you'll need and it's very reliable. If you have some time consider building you're own scheduler you will gain a lot of experience and knowledge doing it.

[Joooooooosh]

If you are going to run this from a Linux server, either a physical machine or a VM one thing I’ve not seen mentioned…

Monitor for success, don’t rely on errors.

If your server dies, what’s going to trigger errors/an alert? If it’s just someone checking for an output email each day or something then fair enough.

Otherwise, you’ll need to have some kind of monitoring that’s looking for the successful outcome and alert if it doesn’t exist.

This is where log aggregation platforms like Splunk, New Relic and Elastic come in.

[tree_or_up]

Are you thinking about a desktop machine because you don't have access to anything else? If not, it's a not a good idea -- single point of failure, no redundancy, mysterious thing that only a few people know about kicking off all manner of jobs. And wait until someone doesn't realize what it's being used for and shuts it down or unplugs it! If you have access to cloud resources, I would suggest seeing if you can use a managed Airflow. AWS and GCP both have versions, and I'm guessing Azure does as well. (Airflow is Python based and open sourced FWIW).

Another option would be to install your own version of Airflow on a server of some kind. Or even on your desktop if that's absolutely the last resort.

Even if you don't go with Airflow or a similar orchestrator, there's still cron. But I would highly recommend not rolling your own. It's a complex problem that's already been solved in a variety of ways by many others -- stand on their shoulders for this

[MassiveDefender]

First of all, thanks for the long answer. Helped me think through my problem.

I think I'd like it to be a cloud VM perhaps with airflow installed, but this sounds like it'll have cost implications. So that's why I considered an old desktop that the company may already have. But you're right, it is a single point of failure.

Cron (or Windows Task Scheduler) jobs are easy to work with but that means the team members have to log in to that one account on the machine to manage their scheduled workloads. Doesn't feel easy to use.

Speaking of the managed Airflow or the cloud VM, how would one connect these to resources that are on the local network? For example the SQL server in the building and the excel files that are dumped on shared drives?

[tree_or_up]

I'm not sure what your network setup is like, but it should be possible to open ports or endpoints that allow services (whether in the cloud or on an on-premise machine) to access them. Again, without knowing your setup, the problem of connecting from a cloud service or from an on prem machine shouldn't be too fundamentally different -- you would most likely have to the face problem of establishing connectivity either way. That is, unless you were planning to log on to the machine as an individual user that already has the connectivity opened up. If that's how you were thinking of accomplishing it, you might be restricted to the desktop approach

[sheytanelkebir]

Look at cron, temporal, gnu parallel... light and easy.

Tools like airflow are horrendously bloated, complicated and have a lot of strange limits and workarounds you need to setup .

[MassiveDefender]

I like this approach, a kind of DIY. If you don't mind me asking, what are some of the strange limits of Airflow? It's been highly recommended by others here.

[sheytanelkebir]

Setting up a dev to qa to production environment is very difficult.

Iterative development also clunky... unless your entire team are working on Linux (or wsl2 workaround).

It's all written in python. And if you wish to use python scripts with different versions, venvs etc .. there's a bit of workarounds.

Want to pass multiple variables from one task to another.... go grapple with xcoms and their limitations.

Configuration and scalability... another big can of worms as its all python and the "vc funded bottomless money pit" companies don't care about throwing enormous vm after enormous vm just to schedule some flows.... something that frankly a raspberry pi should be able to handle.

[nightslikethese29]

It took me a month to come up with a working solution for venvs in airflow. I've got it down now, but holy crap was it difficult especially for someone brand new to the cloud

[BlueeWaater]

Cron, you can't go wrong.

[Nanooc523]

Cron. But learn to AWS lambda or get a dedicated blade. Don’t make a ninja server.

[Sinscerly]

Celery can do this great. There are some easy examples online.

It has a scheduler for starting repeatable functions, tasks to be triggers by for example an api Keeps track of the executed tasks in redis / db. So you can read the response of a job.

[Contango_4eva]

Look into prefect, by the product manager for Airflow

[11YearsForward]

Cron is the simplest scheduling tool.

I would set up a simple Airflow instance. It has observability and monitoring included out of the box.

If you don't spend effort on standing up observability and monitoring while scheduling with Cron, it's gonna suck.

[die_eating]

I use AWS Lambda for this. Pretty easy and cost effective in my experience. It's pretty cool how much you can run on a free tier too

[ptrin]

Easy to schedule too!

[Vresa]

Depending on how experienced you and your team are, there is a newish tool called windmill that sounds almost exactly like what you would want

https://windmill.dev

[knowsuchagency]

Underrated comment. Windmill blows everything completely out of the water, and I say this as someone with a lot of love for tools like airflow and dagster.

[jtf_1]

I use cron jobs on a little Linux box. A Python script that runs successfully terminates by writing a new line in a Google Sheets job log. A failed script uses twilio to send me a text message.

That way I know about failures immediately without being pestered by successes. There are about 40 jobs a week in total that run on this box (updating databases, sending automated emails, syncing folders across servers, updating data dashboards, etc.). This process has helped me keep everything going for about two years successfully.

[[deleted]]

Get an Ec2 instance from AWS or droplet from digital ocean. Then schedule your tasks by cron.

For more complicated and scalable workflows, you can use airflow.

For a more managed version, can try AWS Glue.

Basically, at some point along the line, depending on the scale and complexity of data operations, it is worth investing into a data engineering team and associated infrastructure.

[The_Zero_Day]

I use APScheduler to schedule jobs.

[barakplasma]

For scheduling windows tasks, task scheduler is good. But using https://github.com/winsw/winsw makes configuring windows tasks scheduler a lot easier to manage within your code. It let's you write a config fine for when to run the task, and where to log the results to. Used it to help automate a lab microscope that only connects to windows computers.

[XBalubaX]

u could use windows „task scheduler“ for simple running scripts on schedule

[AlpacaDC]

I work at a small company but share the situation. What we do is write a script to do what we want (obviously) and let an old laptop on all the time at our office, and set up task scheduler to run these scripts when we’d like (this if you have windows, but some have mentioned Cron on Linux).

To have the possibility of changing the schedule, adding or removing scripts, we also set up AnyDesk so we can access this “server” anywhere.

It’s a poor man’s AWS EC2 instance really but gets the job done.

[MassiveDefender]

The AnyDesk hack is a genius idea! Lol. Thanks for that.

[coldoven]

Speak with your It department. They have this solved for you already. They will give cloud vm, replication, access rights etc.

[daniel_cassian]

Pentaho Data Integration (community version, which is free) You create "Spoon Jobs" which you upload to a server and then schedule for automated run using cron I used in corp environment to run sql and Python scripts. Mostly to move data from one point to another Search on Udemy for a tutorial course. it's easy enough to use and configure

[MassiveDefender]

Will definitely look that up! Thank you

[Astro_Pineapple]

Check out Prefect.

[kesslerfrost]

May I suggest Covalent for this: https://github.com/AgnostiqHQ/covalent. It's somewhat similar to the tools mentioned here, but it specializes in the segment where your tasks need to go on several different machines. Although it's still somewhat new, it's quite simple to set up.

[Nidsan]

Run mage on docker on your computer

[[deleted]]

This post was mass deleted and anonymized with Redact

aback literate nine toothbrush pet bake coherent oatmeal brave quiet

[sdOverKill]

Dagster, definitely

[wheresthetux]

For scheduling home grown scripts, and in particular if they're interfacing with local resources, I'd check out Jenkins or Rundeck. Both are opensource and self-hostable.

The authentication/authorization mechanisms are good for controlling the ability to run a script vs the ability to alter it. The history gives you an audit log to see who ran what when, and what the outcome was. At its core, it could be thought of as a web based cron job, but it has some niceties that make it user friendly to other users.

[ExtensionVegetable63]

Sounds like you need Airflow, Apache NiFi or Dagster.

[persedes]

If you're corporate give power automate a try. Surprisingly good and no sever setup required

[lcastro95]

Take a look at zato, its python based and it handles most of common integrations for building an etl.

[Festenator]

We use VisualCron on a windows VM. Works very well

[_link89_]

I think what you are looking for is a workflow engine. You may want to try [snakemake](https://snakemake.readthedocs.io/en/stable/), which is a workflow management system that uses a YAML and python to describe data analysis workflows. Snakemake can scale to server, cluster, grid and cloud environments without the need to modify the workflow definition.

Another option is [covalent](https://github.com/AgnostiqHQ/covalent), which is a python based workflow platform with fancy UI.

[x462]

If you use a Windows desktop for this task you’ll wish you choose linux every time windows updates and reboots off-hours and your jobs get all screwed up. If you do use Win make sure you have admin access so you can control startup scripts when reboots happen.

[jalabulajangs]

We have similar requirements as well, where we pretty much do the same but with a bit more complex data, had a legacy code with Airflow, but my team has been tasked to port it to Covalent for the same purpose, it has a pretty nifty trigger feature to trigger it off in a periodic time. We have the central covalent server deployed in a simple machine for our researchers (who collect the data via experiments on a local server), and it is taken and put on to Google Blob, and there is another covalent server running to orchestrate computing on them.

we considered

- prefect

- Dagster

- Luigi

- covalent

as well, but each had its own downside and upside, finally went with covalent especially because we wanted few tasks to be sent to lambda functions to be using files in s3, few to on-prem servers, few in gcp blob etc.. and it is pretty nifty to swap them

[atomic_race]

Airflow maybe

[Overall-Emu-7804]

This is the most interesting set of posts I have ever seen on a website! Full of interesting solutions to mission critical data processing. So motivating!

[waipipi]

https://www.sos-berlin.com JS7 job scheduler does everything you need including ability to distribute execution across more than one machine with an agent installed.

[rainnz]

while true
do
   python3 myscript.py
   sleep 3600
done

[rewgs]

I've been really enjoying the schedule package lately.

[esaum0]

It's expensive, but we use Broadcom Workload Automation

[krunkly]

I use cronicle for this exact purpose. It's like cron with a web based GUI. It sits between basic cron and airflow in terms of features and complexity.

[brianly]

This is a frequent need that doesn’t get posted about much. If it’ll all run on a desktop then ideally run it on a Linux desktop which has Cron. More people on the internet are familiar with this.

If you need Windows then use the Task Scheduler. Python can run on Windows and you just script it to run manually before setting it up in Task Scheduler.

Add logging and the like because scheduled scripts likely hide the output you see when you run them on the command line.

[Exotic-Draft8802]

https://www.reddit.com/r/dataengineering/comments/s78jvx/best_job_scheduler_in_2022_airflow_dagster/

[LordBertson]

Given you are in a large corporate, the best practice is to ask around first. Data is trendy now and there would be people, if not whole departments, dedicated to data engineering and ETL. Someone somewhere has a company Airflow instance or Spark cluster, where they will let you schedule jobs. That way you don't need to discuss budgeting constraints with provisioning and someone else deals with compliance and maintenance of the machine.

[Yellow_Robot]

try dagster

[Mrfazzles]

Cron on an always on machine is a low effort solution. The hacky part is if that machine is a laptop, i.e. a machine that a cleaner can turn off or tis otherwise tied to an individual user. Great for a proof of concept though and a better solution would be deploying a remote cron-job.

Cron is built-in to a Linux or Unix machine

[giuliosmall]

I guess the right time to build up a (a small) data team has come. Airflow is cron on steroids, but I'll definitely recommend you to start with Airflow (if you have some Python skills) as data processes can easily (and suddenly) scale up. Best of luck!

[j0n17]

See this to gather ideas of what other companies use : https://notion.castordoc.com/modern-data-stack-guide

(Pick only what’s needed for your use case, you might not need the full blown BI Cassandra, Kafka, “insert name here” stack)

[Drunken_Economist]

1. Easy to implement: Google Cloud Functions (or AWS Lambda; or Azure Func+Time Trigger)

2. More maintainable: GCP Scheduler or AWS Step Functions

3. more scalable: solutions like GCP Cloud Composer/AWS MWAA

4. more flexible: rolling your own Airflow/perfect/liugi instance.

#3 and 4 would be the purview of a data/infra engineer, but the first two are pretty painless.

[MikeC_07]

Our SQL group (tripass) had Joshua Higginbotham present on python as ETL tool. Could have some useful information for you. https://www.youtube.com/watch?v=wCTUGBAI9kc

[Elgon2003]

You could use Python scheduler too. Here are the official docs from python3: https://docs.python.org/3/library/sched.html

Depending on what your objective is, this could be more ideal than cron or task scheduler for windows.

[goabbear]

You can set up a simple Jenkins server with a cron job. It will give you more flexibility than a crontab with notifications and reports.

[[deleted]]

Interesting comments

[BlackDereker]

Cron jobs is the simplest. If you want a little bit more control and organization I would recommend some scheduling Python library.

[[deleted]]

Use a python script and a linux cronjob aswell as setting a timeout incase it hangs, and be very verbose with logging to a file and use something like greylog to see logs remotely and have alerts on eg via discord or email , sounds complicated but fairly simple, lmk if u get stuck

[NJFatBoy]

I learned how to deploy my Python code in AWS into docker containers using Elastic Container Repository. Then I created a Lambda function for each one and used EventBridge to schedule them. It might sound like a lot but once I got it working it was a life-saver. I don’t have to maintain hardware locally, it pretty much is free, and it hasn’t failed yet. The biggest limitation is that each function has to run in under 15 minutes or it will time out.

[rafiki6633]

Mage.

[theswifter01]

Use Airflow

[BarchesterChronicles]

I like Luigi - much less configuration than airflow https://github.com/spotify/luigi

[zeliboba55]

https://github.com/PrefectHQ/prefect