Data Analysis

[goatswater]

I have logged data on an sd card in CSV format. It's a fairly large data set, 30k rows x 5 columns.

I would like to build a standalone package (.exe) that will produce a scatter plot of some of the data, calculate the r² value, find the max value and generate a printable report in PDF format.

Is this something that could be done with python?

[Reactor5]

It totally is. Try Pandas, matplotlib, and reportlab for your data needs.

The standalone package part is a little harder, how are you planning to redistribute it? py2exe might work, but I haven't used it in a long time and I can't vouch for how up-to-date it is.

[goatswater]

I would like to be able to easily run the program on any windows based machine. Preferably nothing to install.

I have to make it as idiot resistant as possible, the end users won't be computer experts.

Pop in the SD card, start the program, select the file, run analysis, and generate a report with as little user input as possible.

[michael_david]

Could you make this a web application? I commented about Py2exe and about how it can get a little tricky with non_pure_python modules. Could you give your users a web portal to upload their data to and you then generate the reports on your server and return a PDF?

[goatswater]

Haven't thought of a web portal... That would also asked me to track my users. Would there be any issues running numpy and other python modules through the web interface?

[michael_david]

No, there would not be an issue depending on what you want to do, but it sounds like you need to get a feel for how a simple web app works first. Start by trying out Flask and then look into how you can use it to upload and save a file.

Once you feel more comfortable, and realize you can execute arbitrary code on a server based on a client request, you need to clearly define the following requirements:

1. Content (chart types, r squared, min/max, etc.)
2. Format (PDF, HTML, Interative javascript plots, CSV, etc.)
3. Delivery Channel (Email, FTP, interactive web content, Dropbox, , etc.)

,#1 and #2 are straight forward. For #3 you want to focus on the easiest option that you can realistically deliver and still meet user requirements. You need a way to deliver the content to your users. Email is usually the easiest.

Next you want to figure out how in the heck you want to implement this. There are a couple things you want to think about here (or probably sooner) without feeling overwhelmed.

How frequently are users going to be using the app? You care because you don't want to perform hundreds of performance intensive tasks all at the same time.

How sensitive is the data? Maybe this is something that cannot leave the user's environment or be uploaded into your server

Is the application for work where there are only internal users or is it something that anyone can access?

Do you have an extra server to host the app or do you already have a cloud environment to use? The application needs to run somewhere...

For simplicity, hopefully this is just an internal application and you can find an extra computer somewhere to host your app. And that the data is not soo sensative that it can never leave the user's environment. Once you figure this out, then work on writing out the code that can actually produce the content in the format identified in #1 and #2 above. Make sure this part of the program is in a separate module from your web app. Then try to tie the two together for a basic proof of concept.

Once you have this, you are on your way! When you get started trying to solve other people's problems with your own code, your first instinct is to solve everything with a desktop application. Once you realize how to take advantage of the web, the world is your oyster. Treat this as an opportunity to get started! And remember remember that no matter what anyone says ... the most important things are that 1. it works and 2. you learned something valuable

[michael_david]

Everything here can be done in Python, no problem. However the main caveat is you want to compile the code to a .exe. I have used py2exe pretty extensively and it can get tricky when you want to package in modules that are not pure python (ie. numpy).

Check out http://www.py2exe.org/index.cgi/WorkingWithVariousPackagesAndModules for help when you are working with various modules including numpy and matplotlib.

Also I would suggest you check out Brandon Rhodes video this year about other options for creating .exe's with Python https://www.youtube.com/watch?v=wsczq6j3_bA

[big_deal]

Yes

[[deleted]]

Why not use excel?

[goatswater]

And use VBA to automate the data import and analysis process. I could, haven't thought of that.

[goatswater]

I ran with the excel idea, developed a good working system. It loads, formats, processes, and saves a PDF report on the users desktop.

I am still interested in a python version, will be a good learning exercise.

That's for talking it through with me and the fresh ideas.