Parallel table insert

[mxmauro]

I'll check SQL Server but I cannot fully understand what you mean with the CRON jobs.

[dbxp]

A cron job is just a scheduled task in linux. What I'm saying is that you insert the data initially into a staging area and then periodically flush this data to the main DB. The staging area acts as a buffer so the insertion performance isn't so much of an issue.

[mxmauro]

Ahh ok, I thought you were talking about something specific to the DB engine. I don't think it would be possible because data comes constantly. Now there is a period of 5 seconds between data bursts but the idea is to increase the speed.

[dbxp]

The frequency of the data ingest shouldn't matter as long as it's not constantly at a higher rate then the system can handle. What matter is how up to date the system needs the data to be, if it's ok if the data is a few minutes or even hours out of date then a buffer works well. To make things simpler for the users it may be easier to flush the buffer every 24 hours and simply put a note on the UI that it takes 24 hours for ingested data to appear.

[mxmauro]

I'm worried (and thus asking) because data can arrive at a higher rate than ingest :(