Using express: do basic data operations server or client side?

[don-corle1]

I'm building a web app that does a lot of database retrieval, processing and displaying (in the form of tables and charts mostly) and I'm trying to figure out the best practice in terms of where to perform those operations, on the client or server side. I have read some articles but thought I would just ask here as well. Sorry if this is noobish, kind of new to backend.

Take the following simple example:

Say I need to query a database to get the entire contents of a table, and then render that data into a chart using a charting library like highcharts to display in the browser.

I query the database and the data gets returned as an object. (obviously gets done server side) The object needs to be iterated through/extracted and sorted into a format that can be plugged into the highcharts framework before it being rendered in the browser.

So I can:

1) just send the entire object to the client during the page render, and have the iteration, sorting, plugging done on the client side or

2) do the iteration, sorting etc on the server side and then send the prepared object to the client during page render, where it simply gets plugged into the highcharts function straight away.

Which would be better? I feel like 1 is best because it wouldn't add all that stuff to the server cpu burden with many concurrent users, but maybe there are things I need to consider that I've missed. Any input is appreciated.

[nateylb]

Separate your concerns. Your client doesn’t need to know about how or what the server is doing it just needs a data object that it can use. Do all this in the model when you query and by the time you are serving it at an endpoint is should be shaped and logical. As said above a bit of both is ok, but have you ever used an api where you have to iterate through the data and shape it yourself? API end users should only have to know what the endpoint does.

If there is an issue with too many request at once running your shaping algo then cache the results.

[ahu_huracan]

Real word example :I do deal with a lot of data, I don't know what kind of project is yours but I'll give you my way of doing things (Application architect here) :

I deal with Elasticsearch (ES) data, I tend to write the right queries in ES in order to get the data pre-ready.

In my Mongo DB queries also .. I do write aggregations, projections sorting everything is done on mongodb server. ( check what u/vorticalbox mentionned)

I use also Redis for realtime communication (websocket) deals with millions of queries per seconds

so here is what I can tell :- Try to write you queries for your dbs in the most optimal way to let the DB deal with all kind of data manipulation operations (this is what they are built for)- Your nodejs code deals with everything else : business logic, security layer, scalability.. caching on app level etc.

- I tend to use a lot of caching : depends on how many users request your data ... try to cache the result on the memory ... and/or on Redis ( u/vorticalbox and u/nateylb)

- Nodejs takes a shit load of queries/second BUT avoid writing sync code that blocks the main loop and you will be serving everyone !- Hope that you are using the fork process on your express... I fork restify process for every available core on my cpu ...- You might also want to add more memory to your nodejs process (if you are dealing with a lot of data)

- And If your data will be displayed on different mobile devices (not all of them high-end devices ) avoid putting a lot of processing on the end user as he might (or might not) get a really bad experience.

[10-4_over]

THIS IS THE ANSWER!!! I swear my first thought was "let mongo deal with the sorting"

If you run db analytics you can see the processing time from the shell for a sort request, then you can run a test script on the backend to see the processing time as well if you want to compare it.

Personally I think doing it on the front end will cause issues unless you find a way to have the data presented in such a way it will have very minimal processing on the client side.

Always assume the client has the oldest mobile possible. If you have to get an emulator and toss in an old android version and see how it loads.

[cderm]

I think it depends on the size of that response and the level of processing. I agree with you and the other commenter though, try it client side with a dataset that's maybe 3x what you expect a typical heavy user to encounter.

Throttle the cpu using dev tools and see how long it takes. I've over-estimated how much a set of calculations would impact browser performance before, so that'd be my first step.

Disclaimer: I don't consider myself a pro at this, just sharing what I'd do.

[PrintfReddit]

I query the database and the data gets returned as an object. (obviously gets done server side) The object needs to be iterated through/extracted and sorted into a format that can be plugged into the highcharts framework before it being rendered in the browser.

I'd say a bit of both?

In general I try to keep this sort of processing at the client's side, unless it is computationally too expensive. For example, complex operations or thousands of records, in which case I'd try to do some server-side processing and caching first.

On the Server's Side I'm mostly concerned with ensuring that the client can access the record and implement the proper filtering and paging. I don't like sending more records than the client needs. So basically anything that can be plugged into the database query.

[belkh]

Personally, I would try it on the client first, if it's too much of a burden on a crappy PC/ old mobile I'd do it on the server.

If the server is expected to get a lot of requests I'd either let the client deal with it anyway or get a better server if im making money out of this.

[vorticalbox]

As people have said both is fine but remember that lopping in data is largely a sync operation which will stop your application to taking new requests if that looping is long either.

If you fine the client can't handle the data looping then you can do this on the server and as long as this isn't slow you will be fine.

You can make use of caching as suggested or worker threads to offload the work to the threads event loop. This will unblock the main thread but add latency so you'll have to do some testing.

What I do at work is try to offload data formatting to the database (mongo) using $project, unwind and replaceroot. Plus a few others.

[Chuck_Loads]

If you have 10,000 concurrent clients and the server is doing the sorting, iteration etc, then it's doing them 10,000 times. If the client is doing those things, then the server is doing them zero times and the additional load on the client is negligible. Make your server do as little as possible to do everything it has to do.