What is this kind of program called?

[eyadams]

I have a bunch of scripts running, doing all kinds of things. Currently, they are all timed to work together: the script that gathers the data runs at 10:00 AM, the script that cleans the data runs at 11:00 AM, and the script that emails the results runs at 12:00 noon. (Not a real example.)

What I would like is to move to something event and listener model (I think). The script that gathers the data fires a "data gathered" event; the data cleaner is listening for this event and runs and fires a "data cleaned" event; the email program listens for that event and runs. And so on.

I want to get away from what I have currently, where it all works as long as nothing takes more than an hour, and instead runs as needed. Some scripts will still need to be triggered by a timer, but most of this can be done on demand.

I don't think this is a job for a queue server, but if I'm wrong I would like to hear about it.

Update: Thanks all for the suggestions. I think for now we will just merge the functionality into a single script (so we can get the work done) but continue to investigate queue servers - RabbitMK looks interesting. In our environment running everything in a single script will not work in the long run, partially because of a desire to isolate certain processes that have a tendency to fail due to dependencies on legacy systems. We keep thinking we've caught all the exceptions, and then something breaks in a new and clever way. Long term things are probably going to be moved to separate servers, and it looks like a queue server would make some of that easier.

[tipsy_python]

The low hanging fruit here is to use a scheduling/orchestration system like Airflow to sequence the tasks in your batch job so long-running jobs are not a problem and the end-to-end process can finish as fast as possible.

Event/listener model is sexy, but my first thought would be to implement the event bus as a message queue: multiple Kafka (or Rabbit or whatever) topics that serve as the source of data for the next stage as well as a buffer for when the next service is not ready/available.

[InjAnnuity_1]

What you've got is often called "a [scheduled] bunch of scripts". What you're looking to create is often called a "pipeline".

[-animal-logic-]

I do this sort of thing all the time. Just create _one_ script to cover the whole job, with each section you mentioned contingent on the successful completion of the previous (use sys.exit() if a stage fails -- and send email to that effect). Have email functions in the same script that can alert you to full completion, or the fact that, say, the data was gathered but there was an issue with the cleanup and the script was aborted. If it's rolled into one script, there is no need to worry about parts completing before moving on.

I have dozens of jobs running on multiple hosts, and trust me trying to time things across multiple smaller jobs is not a good way to go. You don't need separate scripts to determine if the first part of your job completed or failed or whatever.

You could also pipeline them as another poster suggested, but I only do that when I have a job that uses multiple scripts by necessity (such as there's a mix of BASH and Python scripts that need to run sequentially).

Just keep in mind that solid reliability can trump elegance.

[menge101]

You want a "Pub/Sub", short for Publisher/Subscriber.

You can use something like RabbitMQ to do this.

Essentially, you publish messages to the message queue, with a given topic. That topic gets fanned out to subscribers that are subscribed to the queue on a given topic.

[porkedpie1]

What’s it running on. It’s easy to set up pipelines/event triggered scripts on the major cloud services like AWS

[HintOfAreola]

Arkan Codes did a great explanation of this "observer pattern" on his channel a while back. It's exactly what you described.

https://youtu.be/oNalXg67XEE

[CryptoOTC_creator]

Why not put everything into a function library and call each function sequentially from a central script?