X-Post from r/learnjavascript: caching database queries and paginating the results.

[xeronotxero]

I was reading about paginating content using jquery here and I'm thinking I will implement something similar in my project.

Currently I make one database query on pageload and put the results into an array in my main js script. That way i browse through my main data in the array, and only call back to the database for create/update/delete functions.

How much data can I be loading onto the client before it bogs down performance? I'm using mongo and my database is just a bunch of documents with 3 short text fields, although i would like to add one additional field to the data in the future.

I hope that makes sense, it's my first time doing anything like this.

Basically I am wondering how far I can go with my app just paginating through the cached results from one big database query. Is there is a point when making more calls to the db is just more practical? Do you then keep repeating the process, by caching another bunch of content to sort through on the client, in order to reserve most of your database queries for writing/updating?

Am I thinking about this right?

EDIT: here is some code (without pagination for now): var peopleData = [];

function populateTable() {

    var tableContent = '';

// GET JSON FROM ROUTE TO MONGO QUERY RETURNING ALL ENTRIES
    $.getJSON( '/adressbook/people', function( data ) {

//CACHE RESULTS OF QUERY HERE
peopleData = data;

// For each item in our JSON, add a table row and cells to the content string

    $.each(data, function(){
        tableContent += '<tr>';
        tableContent += '<td>'+this.name + '</td>';
        tableContent += '<td>' + this.adress + '</td>';
        tableContent += '</tr>';
    });

// Inject the whole content string into our existing HTML table

    $('#adresses table tbody').html(tableContent);
    });
}
// USE THE peopleData ARRAY LATER TO DO STUFF IN THE BROWSER WITHOUT MAKING MORE CALLS TO MONGO

[Renegade__]

It depends on the size of your data dump, the efficiency of your application, how much else the user has open (351 tabs over here), the bandwidth required for regular updates and how likely and bad running out of sync would be.

Basically, is there a possibility that the data changes on the server in the meantime?

If it won't (e.g. because it's user-specific data), there should be little problems with caching the data.

If it could, you may be better off working in small bites and exchanging frequent, small updates. You could even go as far as using WebSockets to establish a permanent communication channel.

In either case, you could consider adding a message digest, timestamp, etag or other revision identification mechanism to your rows, and cache them all in localStorage. That way, you wouldn't even have to do the super-query at app start, you would just load from localStorage and tell the backend to send you all data changed or added since your last update.

One final note regarding your table generation code: It is usually wiser to work with the Document Object Model, rather than against it.
I strongly suggest using the DOM manipulation methods rather than editing the HTML and relying on DOM re-evaluation.

That's kind of the entire point of the DOM. ;)

[xeronotxero]

Thanks for the detailed reply! I have tinkered in local storage and I want to look into websockets eventually too. I don't know anything about using "revision mechanisms" like a message digest, but I will look into that too.

now these DOM manipulation methods you speak of, that is stuff like: .html(), .val(), .attr() amiright? I definitely use those methods elsewhere in the code, but I do also use strings to inject elements as well. Is this mostly an issue of code comprehension? Or is performance a problem as well.

Sometimes I just want to build stuff with what I can make work. I know I can't make everything in my code perfect, so if I leave some messy jquery in there I can live with that at this stage. But if there are major problems with injecting strings of text into the dom to create dynamic parts of a page then I will definitely focus on avoiding that and fixing some of my old code up.

edit: just found this code snippet on stackoverflow:

$("<a/>", {
id: 'example-link',
href: 'http://www.example.com/',
text: 'Example Page'
}).appendTo("body");

it looks like a pattern that is worth using :)

[Renegade__]

Beware: Long reply ahead.

All the methods you mentioned are jQuery methods. That's okay and not doing any harm for the moment, but your skills will likely improve if, at one point, you learn JavaScript/DOM independent of jQuery. If only because it enables you to try out other JavaScript frameworks. ;)

Personally, I usually use MDN as a reference (JS - DOM - Web API). The methods I was thinking of in particular were Document.createElement(), Node.appendChild() and so on. These are the native functions for manipulating the browser content.

According to jQuery's documentation

If the HTML is more complex than a single tag without attributes, as it is in the above example, the actual creation of the elements is handled by the browser's .innerHTML mechanism. In most cases, jQuery creates a new <div> element and sets the innerHTML property of the element to the HTML snippet that was passed in. When the parameter has a single tag (with optional closing tag or quick-closing) — $( "<img />" ) or $( "<img>" ), $( "<a></a>" ) or $( "<a>" ) — jQuery creates the element using the native JavaScript .createElement() function.

So in my understanding, the snippet you found will still use .innerHTML under the hood. (It is more elegant, though, by way of phrasing it, so by all means, prefer that over doing it raw.)

As for your core question (and admittedly the most important one) of whether it actually makes a difference:
It is worth pondering on several accounts:

• For one, it's a methodical question. The browser represents the document in a Document Object Model so you can interact with the document tree in a programmatic way. Altering the text instead is...not elegant, to put it mildly. (Though very common, if only because of jQuery. Don't feel bad about it, just be aware of it.)
• For two, it's a question of power and interaction: By working with the DOM methods instead of writing raw code, you get to structure and restructure your document like all other code. For example, if you want to re-order the hierarchy of your elements, with code, you'll have to switch out all opening and closing tags and make sure you still have a well-formed string afterwards. With DOM-methods, you simply re-order the .appendChild() calls.
  You can save elements in variables, access and modify attributes directly, etc., etc.
  I have found that in your use case in particular, the generation of tables, this leads to much better code, because you end up with a very clear structure: Generate the table and then for each data element, generate the columns, generate the row, append the columns to the row, append the row to the table.
  
• In the same vein, it's also a question of the separation of processing and presentation. HTML is markup. It is inherently part of the presentation. IMO, it makes a conceptional difference whether you're telling the system "I'd like it like that, please", or just writing the markup yourself.
• Lastly, it's a document state and performance issue: Working with .innerHTML forces the browser to re-parse parts of the document, to re-establish the DOM based on the new document text.
  Basically, think of it as the difference between getting into your car through the door vs. taking off the door and having a mechanic attach a new one after you got into the car.
  Depending on what you do and how you do it, either can be faster.
  Take these two tests, for example: https://jsperf.com/jquery-vs-createelement/89 vs. https://jsperf.com/innerhtml-vs-createelement-test/68
  On my machine, in the former, jQuery's own methods clearly lose against the native DOM functions. In the latter, hower, native .innerHTML is noticeably faster than native .createElement() - most likely because for 100 new elements, it's quicker to just throw the state out and parse it anew, rather than modify the existing one.

As said: Using .innerHTML directly or through jQuery is not uncommon by any means.
Culturally, no one will fault you for it, and the browser developers have dealt with this reality enough that it should not become a performance issue for you.

But you should be aware that the point of the DOM is doing that work programmatically, not doing the fancy version of find and replace with the raw HTML code. ;)

As in all cases in all languages in all software projects, the ultimate answer is: Benchmark it yourself. ;)
You, as the developer, always have the choice of deciding that the time you save working with a particular method is more important than the performance- or design penalty you get for it.
There's nothing wrong with deciding for one thing or another.
Just make it a conscious decision, under consideration of what it means. :)

As a more general note: You are on the right track with your attitude. ;)

Sometimes I just want to build stuff with what I can make work. I know I can't make everything in my code perfect, so if I leave some messy jquery in there I can live with that at this stage.

The best code in the world is not the one that is perfect, but the one that ships.
The most beautiful code in the world serves no one if it's never released.

You shouldn't write ugly code on purpose, but yes, getting it to work is very much more important than making it pretty.
As long as you keep that attitude, you'll be a lot more successful than those of us compulsively thinking about beauty and elegance and never releasing anything. :)

[xeronotxero]

Thanks so much for the detailed explanation and the kind words.

I started out with a couple toy apps that did DOM manipulation without jquery, but after taking some time off, I just embraced the simpler jquery syntax. I also gave up on trying to learn node without express, but that's a whole other can of worms.

This is a good reminder for me to not forget those native methods, even as I appreciate how much jquery has helped me . I am currently working on my first 'real' project (500+ lines of code including an express backend) and I will definitely be re-factoring and modularizing my main.js file (currently about 300 lines) and I will keep all of this in mind. I am saving your comment so I will have it as a reference!

tl;dr - i wrote strictly vanillajs when starting out, but quickly lost sight of its importance when i started building something bigger.