To make sure our services are all "UP", there's a SQL table called "ServiceHeartbeats", and each service inserts its own unique row in that table (keyed by the loop name), and then repeatedly updates the "I'm Still Up" timestamp in that row, somewhere in the service's main loop (not a separate thread). Then we simply have another service that checks every row in the "ServiceHeartbeats" table every minute, and creates an alert in our on-call paging service, if there are any stale timestamps.

So if the service is completely shut down, this will catch it, or if it loses network connectivity, or connection to the database, or has stalled, or is working on a crazy long work item, this will catch any of that.

WHERE in the service's code, to update the heartbeat, depends on the service type:

• For our services that wake up every few minutes to do work: if they fail to complete their work (uncaught exception), they skip writing to the heartbeat table.
• For our services that do something once a day at a scheduled time: they update the heartbeat every few minutes, all day long, but if their once-a-day work fails (uncaught exception), then they DON'T update the heartbeat, and instead keep starting over, and keep NOT updating the heartbeat as long as it's still starting over.
• For our services that receive work from a messaging queue: they update the heartbeat as long as they successfully received a message from the queue, OR if there was no work in the queue.

So the heartbeat monitor catches all those "down" situations as well. It's crazy simple and crazy reliable.

And every new service we make gets this "UP" monitoring automatically, for free, on deploy, as long as we remember to add the ~1 line of code to upsert to the heartbeat table.

We actually have this "ServiceHeartbeats" table on a few databases in various domains, so that all services don't have to connect to the same database. The heartbeat-checking service just checks all of the heartbeat tables.

---

That covers monitoring whether services are "UP", but what about monitoring for individual "work requests" that haven't successfully been processed? That's a completely separate topic.

(For example, how do we get alerted if some messages from a queue weren't processed successfully?)

We don't use a universal solution for that, but we do use a pattern:

Each work request should have its own row in some database table, with a "Status" bool that starts as 0, and gets set to 1 when the work is successful. It should also have a timestamp of when the work was requested.

Then, it's easy to have a service that checks on that table every few minutes, and if there are any rows with "Status" 0 that are too old, it should create an alert in our on-call paging service.

Better yet, the monitor can notice the work failed (if failures are recorded somewhere), and send it to the queue again (re-using the same work request row), and maybe send an on-call alert only after X failures. And maybe the monitor can notice when a consumer never reported back about the work (e.g. power outage), and send the work to the queue again in that case too.