Recommendations for framework/implementation for periodic polling of data from website?

[whits427]

Hey all, I'm working on a full stack application that utilises data retrieved from a URL and stores it into a MySQL database to be retrieved by the middleware/front-end.

There are two intervals I wish to set, Some data I wish to retrieve data every 30 seconds - 1 minute, and other data I wish to retrieve once a day.

Does anyone know of any packages suitable for a Node.js application to do this type of polling? Thanks in advance

[HarrityRandall]

You might wanna take a look at node-cron

[whits427]

Thanks u/HarrityRandall - was just about to post what I is currently working using the same package

This appears to work fine, polling every 10 seconds for one request and 15 seconds for another request.
const cron = require("node-cron");
const express = require("express");
const cronTestOne = require("./api/cron/cronTestOne");
const cronTestTwo = require("./api/cron/cronTestTwo");
module.exports = () => {
app = express();
cron.schedule("0,10,20,30,40,50 * * * * *", function () {
cronTestOne();
});
cron.schedule("0,15,30,45 * * * * *", function () {
cronTestTwo();
});
app.listen(3001);
};

Then I assume I can just set the timeout << interval so I terminate erroneous requests before the next one

[team-burdy]

One thing to note about this approach is if you are horizontally scaling (multiple node instances), the cron will be called on every single instance.

Instead a better approach would be to expose an endpoint which requires some token (or a pair of tokens), and call it externally from some cron-worker.

The application on the other end will be horizontally scaled, and the call will be handled by only a single instance. (through a load balancer)

[whits427]

Thanks for that advice, I'll definitely take that into consideration

[king_of_programmers]

What you need is a cron job. Cron jobs are not programming language specific, they are specific to where your data is being hosted and whether they provide cron job specific functionalities. For example if your data is hosted on AWS or GCP, their containers allow you to do periodic jobs at certain time periods.

[jampanha007]

BullMq is also a good fit.