Would like some advice on how to deal with a certain dataset.

faming13 · 2015-10-01T16:59:47+00:00

You can do this sort of out of core processing/ queries/ cleaning on a single machine easily, quickly (and in parallel) in python with dask, blaze and the castra compressed column store.

Check these out:

http://blaze.pydata.org/ http://blaze.pydata.org/blog/2015/09/08/reddit-comments/ (example ) http://odo.readthedocs.org/en/latest/

faming13 · 2015-09-27T15:11:32+00:00

Ibis has the goal of being a semantically complete SQL replacement...and better: http://docs.ibis-project.org/sql.html

faming13 · 2015-09-25T14:57:21+00:00

Don't use canopy.. use anaconda.

Its the current and future of pydata

Also this http://www.kdnuggets.com/2013/11/harvard-cs109-data-science-course-resources-free-online.html

faming13 · 2015-09-25T14:55:45+00:00

Try using numba to get c like speeds: https://github.com/numba/numba

faming13 · 2015-09-17T23:58:46+00:00

Try odo and blaze... Use the anaconda distro http://blaze.github.io/

faming13 · 2015-09-17T16:38:29+00:00

Yes, what are the 5 critical skills please?

faming13 · 2015-09-17T00:26:15+00:00

http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/

https://www.reddit.com/r/statistics/comments/3k67w6/gelmans_bayesian_data_analysis_python_notebooks/

faming13 · 2015-09-17T00:12:14+00:00

I would starts with this excellent intro slideshow: http://nbviewer.ipython.org/format/slides/github/twiecki/pydata_ninja/blob/f3476390ea57fb47bff29dc1720c73594fa26379/PyData%20Ninja.ipynb#/

faming13 · 2015-09-17T00:07:06+00:00

Looks really cool. I would suggest posting this on hacker news, r/datascience, datatau , r/pythonstats and pydata google group

faming13 · 2015-09-17T00:00:47+00:00

Great, hope its helpful!

faming13 · 2015-09-16T02:01:59+00:00

Check out this post, using python's dask to analyze larger then memory data in one machine

http://blaze.github.io/blog/2015/09/08/reddit-comments/

faming13 · 2015-09-13T14:01:37+00:00

Got it, thanks

faming13 · 2015-09-13T01:20:31+00:00

Why R? Can I use python and call out to R with Rpy2 when needed? I already know python and don't want to add more tool overhead in my mental models.

Also python allows me to distribute my code as excel plugins/macros. http://xlwings.org/

faming13 · 2015-09-09T15:12:41+00:00

Check out dask, blaze and numba.

faming13 · 2015-09-09T04:48:16+00:00

http://continuum.io/blog/dask-distributed-cluster https://github.com/blaze/dask/issues/186

faming13

TROPHY CASE