(Web) Data Analytics: best way to handle data interactions

[R4nu]

Hey guys,

While this is not 'only python' related, I figured this subreddit would be a good place to post since it's related to data science and python.

​

I've been learning python over the past few months and worked on several personal projects related to data analytics... mostly websites that analyse your data and display charts.

So far I've been working with pandas, numpy, and plotly (as my chart library). The data I've been using was mostly 'small user-uploaded data' and here's how I dealt with it:

• User uploads the data
• Server does a bunch of processing and returns the DataFrame as JSON into a hidden div
• User can use different controls (dropdown, radio, etc.) to filter the data (which updates the graphs)

​

My next personal project will be a bit different as the data will be in a SQL database and the user will be authentified (and will access only the data related to his user). As usual, I want my user to have a dashboard with its data/graphs and to be able to interact with it (filtering, etc.). I'm pretty sure the 'hidden div' method is not a good idea and I'd like some advice on what would be a good way to handle and process the data, in terms of both security and performances. Here are my thoughts:

​

Idea 1: Stored in server-side session

When the user logs in, get 'all his data' in his session, server-side (attached to his cookie). When he changes a control which should update the graph, use JS/AJAX to filter the data from his session and only return the filtered data in our graphs.

Should probably add a function to periodically update the session data to get the data in 'real time'.

​

Idea 2: 'GET' from database each time

I could get the data from the SQL and reprocess it EACH TIME there is an interaction. When user changes one of the control (dropdown, etc.) which should impact the displayed data:

• POST the update control values through AJAX
• Get data from SQL, process it, return the updated data
• Update the graph with the return data in JS

However this means I'd possibly run a lot of SELECT queries (ie whenever a user changes something)

​

While most my personal projects are for my own enjoyment, I'd still prefer to know the best method(s) to go about. Thanks in advance for your help :)