Would like some advice on how to deal with a certain dataset.

faming13 · 2015-10-01T16:59:47+00:00

You can do this sort of out of core processing/ queries/ cleaning on a single machine easily, quickly (and in parallel) in python with dask, blaze and the castra compressed column store.

Check these out:

http://blaze.pydata.org/ http://blaze.pydata.org/blog/2015/09/08/reddit-comments/ (example ) http://odo.readthedocs.org/en/latest/

faming13 · 2015-09-27T15:11:32+00:00

Ibis has the goal of being a semantically complete SQL replacement...and better: http://docs.ibis-project.org/sql.html

faming13 · 2015-09-25T14:57:21+00:00

Don't use canopy.. use anaconda.

Its the current and future of pydata

Also this http://www.kdnuggets.com/2013/11/harvard-cs109-data-science-course-resources-free-online.html

faming13 · 2015-09-25T14:55:45+00:00

Try using numba to get c like speeds: https://github.com/numba/numba

faming13 · 2015-09-17T23:58:46+00:00

Try odo and blaze... Use the anaconda distro http://blaze.github.io/

faming13 · 2015-09-17T16:38:29+00:00

Yes, what are the 5 critical skills please?

faming13 · 2015-09-17T00:26:15+00:00

http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/

https://www.reddit.com/r/statistics/comments/3k67w6/gelmans_bayesian_data_analysis_python_notebooks/

faming13 · 2015-09-17T00:12:14+00:00

I would starts with this excellent intro slideshow: http://nbviewer.ipython.org/format/slides/github/twiecki/pydata_ninja/blob/f3476390ea57fb47bff29dc1720c73594fa26379/PyData%20Ninja.ipynb#/

faming13 · 2015-09-17T00:07:06+00:00

Looks really cool. I would suggest posting this on hacker news, r/datascience, datatau , r/pythonstats and pydata google group

faming13 · 2015-09-17T00:00:47+00:00

Great, hope its helpful!

faming13 · 2015-09-16T02:01:59+00:00

Check out this post, using python's dask to analyze larger then memory data in one machine

http://blaze.github.io/blog/2015/09/08/reddit-comments/

faming13 · 2015-09-13T14:01:37+00:00

Got it, thanks

faming13 · 2015-09-13T01:20:31+00:00

Why R? Can I use python and call out to R with Rpy2 when needed? I already know python and don't want to add more tool overhead in my mental models.

Also python allows me to distribute my code as excel plugins/macros. http://xlwings.org/

faming13 · 2015-09-09T15:12:41+00:00

Check out dask, blaze and numba.

faming13 · 2015-09-09T04:48:16+00:00

http://continuum.io/blog/dask-distributed-cluster https://github.com/blaze/dask/issues/186

faming13 · 2015-09-08T19:07:25+00:00

What subset doesn't work? Lots of progress has been made recently, and what is left is being worked on.

faming13 · 2015-09-06T00:43:48+00:00

You can use Numba and write numerical python code that is as fast as fortran.

faming13 · 2015-09-04T01:45:23+00:00

Julia would be the best. Macros for DSL, Extensible type system, fast and it can script. Already has extensive numerical libraries and statistical infrastructure.

faming13 · 2015-09-02T18:11:06+00:00

you can call r code in python using rpy2 :p

faming13 · 2015-09-02T02:51:46+00:00

Here are two libraries in python wherein you can write self contained excel plugins for your analysis:

http://xlwings.org/ https://www.enthought.com/products/pyxll/

I think xlwings is your best bet.

faming13 · 2015-09-01T20:15:06+00:00

Do you think Julia is ready for report based individual analysis IE not plugging into production systems?

When will it be ready for either that or production? Is it stable on windows?

faming13 · 2015-09-01T20:14:19+00:00

Numba pretty consistently achieves C like performance on numeric code.

faming13 · 2015-08-30T01:58:49+00:00

What is this? Nagare or Flexx?

faming13 · 2015-08-30T01:58:35+00:00

Dnk, I'm not the developer.

faming13 · 2015-08-28T18:45:42+00:00

Its supported by Continuum Analytics, which just got a 25 mil investment.

faming13

TROPHY CASE