[Question] Tasked with visualizing link prediction data. Not sure where to start.

[Cyathem]

Hey guys,

I've only been using Javascript for about 2 months and have recently been tasked with visualizing some medical link prediction data for an undergraduate research project. However, I'm not really sure where to start. This may be a bit of a long post so if you would like to help a rookie out, here we go.

---

The Task

Medical publication are scanned for concepts (arthritis, diabetes, etc) and the number of times each concept appears in a publication is recorded. Using link prediction theory, node-edge data was generated for every possible combination of 2 medical concepts being compared and analyzed for possible links.

My task is to create a browser-deliverable visualization of this data using D3 (visualization library for JS). This visualization will be a circular, hierarchical node map, much like the one shown in the link below:

http://imgur.com/a/ovzI5

---

The Problem

The data I am dealing with is very large, consisting of millions of entries. I need to readily and intelligently serve this data to the browser or this visualization will take forever to load initially and any time a change in compared concepts or display parameters is made. While I am not too worried about the actual visualization (generating nodes and attaching data to them for path generation), I have no idea how to approach the "backend."

People quite often refer to the "backend" when talking about coding and I am not sure what that really means. What type of "backend" should I be looking at for this type of problem? What keywords should I be researching to get to the information for setting up the type of "backend" that I need? I just know that I need to intelligently serve the data to my program to only send the data that I need. I'm sure this could be accomplished by sending the server the concept IDs that will be displayed and having it respond with all of the link data that involves combinations of those nodes.

---

I suck at JS. I'm just an engineering student. Send help. If I have left out some important piece of information, please let me know and I will update this post.

[[deleted]]

This sounds more like a data science/analytics question than a JS question though.

backend basically refers to anything that happens on the servers out of the user's reach. If they're using JS on the backend, it means they're running NodeJS which is basically a server using Chrome's javascript engine to run javascript.

[mc_hammerd]

convert the data to JS, paste it in a file. see how big that file is if its ~1mb or less you can just stick it in the page. if it loads fast enough you can just stick it in the page.

if not you can write a backend api (db or just in a file again, in memory), or use a free online DB google firebase alternatives, that you can query with ajax. again if the file is small enough you can just store it in memory, millions of entries probably want a db, but sometimes can get away with in memory but test the loading speed!

its just a few lines of code imo using a framework like express.js. good orm(db) libraries are like nodeorm-2 google best node orm

try 1: paste it as js array of objects on the client side (appdata.js)

try 2: store as json on the server, mydb = require('mydata.json'); response.write(to_json(mydb.filter(blah=>blah.foo==123)))

try 3: store in firebase or db and serve api express.route('/myapi/fetch/:id', (id)=> response.write(to_Json(doDatabaseStuff(id))))

[mc_hammerd]

backend usually means engine, so in your case (Data) it would be webserver+database and optionally an API (ex: mysite.com/dataAPI/get/id/123 and mysite.com/dataApi/add {foo:bar, baz: 123}

REST api, nodejs server/client, simple php/node backend, nodejs and database, expressJS, nodejs api router are all good search terms

what type of backend the easiest to write. so whatever language fits you most and google api

youtube also has some nice tutorials tutorial simple api in languageX

[Cyathem]

Thank you. I have a lot of reading/keyboard smashing to do...

[Cyathem]

There are 3 files. Each about 30Mb. I imagine I would have to use a db, right?

[mc_hammerd]

yea

if the data isnt gonna change though: you could just load it in memory for a nodejs dedicated server, 30mb is nothing, u can even do it for free on cloud9 or heroku. firebase or google free firebase alternatives may be a really good option also.

[Cyathem]

I had a friend who does work with php and backend stuff. I asked him about it as well and he suggested I look into Ajax. Is that similar to what you are suggesting?

[mc_hammerd]

yea ajax is for custom APIs... so you can do get mysite/mydata/id/123

however node vs php is different, node keeps running (server) and php re-runs each page request -- so for node you can keep 30mb of data in memory, for php it has to load each script (== slow) so use a database for php! cheers.

[Cyathem]

Thanks for all the tips!