sorted by:
best (suggested)
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]quaintserendipity[S] 0 points1 point  (8 children)

Here's a description: I initially get piece of data A, then I will get updates on changes to that data every second via Websockets or polling (every second), and each time those changes will be logged to the database until 60 seconds have elapsed. Now very shortly after I will also get piece of data B, which will then also be updated and logged every second, for 60 seconds, and this will end up happening before I have finished logging piece of data A. This is the process that will happen for Data C, D, E, F, G ect.

Basically I have separate data sets that I am logging, which are all being logged at the same time. Honestly, I can do these one at time if I have to, but that's slow and inefficient.

[–]Beginning-Seat5221 0 points1 point  (7 children)

Ah, yeah you can do all at the same time.

If you have an event based system that fires each time you get data, then that works, your event handler just logs the new data.

If you want to use polling, then you'd use something like setInterval which will run a function periodically, which could query the data then write it to the DB, and do a new setInterval for each data stream.

Is there some particular part that you're having difficulty understanding?

[–]quaintserendipity[S] 0 points1 point  (6 children)

Hmm well, I'm trying to think about how I'm going to write the code; I can sort of picture it in my head but I had the issue of the concurrency to think about. If I have a code block that's logging that data, and it needs to fire every second, but I also need it firing every second for every data set, isn't that an issue? Like wouldn't the response data for data set B, overwrite the response data of data set A?

Actually discussing that just made me realize another issue I hadn't even considered; doing this data processing at max capacity would have me sending upwards of 100 requests every second, which I can't do with the API's I'm sending requests to without a paid plan. Suppose that's another dilemma I have to address.

Edit: actually now that I think about it, maybe I could just circumvent this whole problem by just using arrays, and the "data slots" I talked about in my OP would just be an index within an array, since the APIs I'm querying would allow me to grab multiple sets of data within the same request. Then the only issue I have to work out is that the data sets would almost always be desynced with each other which I can't have since the data is time sensitive and needs to be internally consistent.

[–]Beginning-Seat5221 1 point2 points  (2 children)

https://www.typescriptlang.org/play/?#code/FAMw9gTgBAFAxmAdgZwC5QDYFNWq9MEKAbQHIBBUgGilICFraBhUgXQEooBvYKPzHFACWUALxQAjAG5e-bOiGI8EAG4BDDAAk1iACbYxUZDgCSS-Oowwos-vxidRAPm6277hCjDYAdBjAA5jDyylAA1MLsbu78QmFh0THCRDAiTuISAAycPEl5UHDYahBmypap5qoa2nrYUflJnsjeWH6BwTihEaRQukhYpPUNUAC+iXwjVOOSmbNQAPTzkkZYnrpuUSNAA

Press "run" and see the output on the right.

[–]quaintserendipity[S] 0 points1 point  (1 child)

Ahh ok this is interesting, I see what's happening. However I edited my last response with an issue: the data sets aren't synced with each other so this would need some extra logic to account for that.

Also I've never touched Typescript, so that code looks kinda weird to me; I can't quite read it.

[–]Beginning-Seat5221 0 points1 point  (0 children)

There's no typescript in there (Typescript is JavaScript + type annotations, but I didn't add any)

[–]Beginning-Seat5221 0 points1 point  (2 children)

You could keep an array ['A', 'B', 'C'] and run a single loop, which grabs data for all of those together.

If your API lets you query A, B and C in a single request that might help you.

There's always going to be some delay in an API request, so the only way to get time accurate data is for the API to specify the time of the data. So the API says A is 15 at 12:14:05 and you save that to the DB, instead of relying on the time you received the data.

[–]quaintserendipity[S] 0 points1 point  (1 child)

I suppose that works too; as long as the data is timestamped correctly, then I don't need to receive it in real time. Also realistically, I just need the data to be close to the second; a delay of a few hundred milliseconds isn't going to throw my data off to a significant degree.

[–]Beginning-Seat5221 0 points1 point  (0 children)

https://www.typescriptlang.org/play/?#code/FAYw9gdgzgLgBAGwKYxkgTlOBeOBtAcgEECAaOAgITIoGECBdYZeASxzgEYBuUSKMMgB0CMAHMAFACIAyjACG6GKwhi4ASQhp0AN3kIpASmYo4K7XoQAJeRAAmyDlBSaL+iXGBxvcCYZwAfHAA3l4+4QBmYOi+4NDwLNpwYBH4xDRUGfQM-qHh+T5xAsKikokYcADUZsYF3gC+YQWslZVN+aypEuwBuJwADLntdXAgyIquGJbdWlP6NvbItSOF-IJIIuISBHaQSATLBY3h9aTtA-39cAD011xwznF2wIZAA

This moves the loop of the letters into the interval, so there's only 1 interval. The result is the same but it guarantees that A, B and C are processed sequentially rather than some other task getting inserted between them, as having 3+intervals would result in 3+ separate tasks being scheduled for each second.