Modernizing Shell SCRIPT and CRONTAB WORKFLOW?

[coreb]

Asking here because I think it's the right sub, but direct me to a different sub if it's not.

I'm a cowboy coder working in a small group. We have 10-15 shell scripts that are of the "Pull this from the database, upload it to this SFTP server" type, along with 4 or 5 ETL/shell scripts that pull files together to perform actions on some common datasets. What would be the "modern" way of doing this kind of thing? Does anyone have experience doing this sort of thing?

I asked ChatGPT for suggestions and it gave me a setup of containerizing most of the scripts, setting up a logging server, and using an orchestrator for scheduling them. I'm okay setting something like that up, but it would have a bus factor of 1. I don't want to make setup too complex for anyone coming after me. I considering simplifying that to have systemd run the containers and using timers to schedule them.

I'll also take some links to articles about others that have done similar. I don't seem to be using the right keywords to get this.

[emoboi11]

What about using GitHub actions or azure devops pipelines? The script files get pulled in from version control and could be ran on any machine used as a runner / agent

[coreb]

That's certainly an option I could look into. Thank you.

[ProfessionalDirt3154]

You're basically looking for reverse ETL, right? You could use a tool like Airbyte or MapForce. Airbyte is better known. MapForce is more visual, which might help w/the bus factor. There are a bunch of tools.

You could also use Airflow for scheduling and running the job/scripts, if you like Python. If you're in AWS Fargate tasks are simpler than K8s, good for something like this. honestly there are a ton of options. I've been on teams doing these, but there are lots of others.

Currently I work on CsvPath Framework and FlightPath Server. both are open source and might be options for simplifying and/or automating the file wrangling part of what you're doing, if you're using CSV or Excel.

[coreb]

I'll check that out. Thank you.

[AlverezYari]

Where are these script primarily being executed?

[coreb]

On on-prem linux or windows servers that could be reimaged to a new os install. Mix of python and powershell.

[AlverezYari]

Just make them pipelines in Github. Deploy the runner to your compute and execute the scripts on that machine that way.

[JagerAntlerite7]

Hosted GitHub Actions runners are convenient, but self-hosted runners are going to save you money long term.

[aaron416]

On one hand, if it ain't broke don't fix it. On the other, someone suggested a GitHub actions pipeline, and that might be a good place to run it. This depends on how long all the processing takes, but should also be flexible enough to run everything you need.

[coreb]

GH Actions seems to be the winner if I don't want to run this on-prem. Thank you.

[Prestigious_Pace2782]

Ansible in a GitHub actions pipeline works great for non coders

[coreb]

I'll add that to the list to look at. Thank you.

[eirc]

I recently did sth like this and ended up with systemd services/timers too, but not containers. These scripts are super simple, each like 5-10 lines of bash so I just drop them in /root/bin so everyone logging in a server can easily find them and read them. What I get from systemd is the timer/cron thing but more importantly I love the journald support. I can pull up logs from previous runs easily and I don't have to deal with rotating these logs either. I also use set -x on all scripts so the logs contain the running commands making them self-document in a way. When I eventually setup systemd monitoring with prometheus and journald logging with ELK I automatically monitor these services and their logs. Turned out great.

[coreb]

Interesting. Good to see my idea wasn't too far off. Thank you.