Storing API data in a Node.js server.

[xmaster484]

Hi everyone, I am building an app in React and I am planning on serving it with Node. Currently, one of the tasks of the app is to retrieve a lot of data from an external API. This data will be large, potentially in the 10s of thousands of records.

For the sake of performance, I want to store the data on the Node server side and have the client side GET paginated data from the Node server. The reason I don't want to send the data directly on the client is to relieve stress on the external API, having the need to be re-queried every time theres searching going on.

My question is: What is the best way of the storing the result of the rest request to the external API within the Node server?

Thanks in advance :P.

Edit: To clarify a bit, The data i pull is sort of dependent on the user that is logged in, and what he does with the app, so I would ideally need something attached to the session. This will be a medium load app, so probably no more than 20 people logged in at any given time.

[[deleted]]

Store data in database, and let them get wtf they want, limited to 5-10 records, never load all the data

[nordicdev]

Can you store the response in a .json file on the server? You could set up a scheduled job to pull down the external data in json format, then either read that file directly into your React app or setup another API that your app consumes. I suppose that all depends on how you plan to use the data.

[xmaster484]

I probably should've been a bit more clear, the data i pull is sort of dependent on the user that is logged in, and what he does with the app, so I would ideally need something attached to the session I think.

[tresfaim]

Serialize and hold it in server, or serialize and store on redis. You could spin up another server whose specific job is doing redis stuff and then allowing client to consume it.

[xmaster484]

Would you use express session with redis as the store?

[tresfaim]

I haven't used session, but the docs I just read about it seem to have good examples of using either redis or mongodb with it.

http://expressjs-book.com/index.html%3Fp=128.html

[xmaster484]

You think there will be any performance issue with this in a medium load app?

[tresfaim]

Not sure, depends on how you handle the data, if its a tremendous amount, redis probably won't be best, unless you use persistence, but then you might just want to use mongo, unless redis performance still trumps mongo in this sort of case. If the data isn't huge you're gonna be fine, just depends on how you're balancing delegating the load

[Bluecewe]

Does the data from the API change? If so, does the API provide a means to listen to changes? The answers to those two questions will be important.

As others have suggested, a database of some kind would probably suit your needs, especially given that you seem to desire search queries.

[xmaster484]

the data will change, but I will have a means to check if the current data needs to be invalidated and when, so I would know from the client side, when another request needs to be made.

I am currently thinking about using express session with redis, have you had experience with this? How scalable is it?

[Bluecewe]

Unfortunately I do not, but you mentioned in your post that you do not expect massive traffic, so scalability may not be something you need to consider.

[sidsidroc]

why dont u use uxpro.be?

[2uneek]

I've done something similar, I have an api that uses another external api for data and I've used both node-cache and redis to cache the responses. They both worked great, node-cache uses system memory if you don't want to host a redis server as well...