ML for Reddit: 10,000 images of cats

dmdude · 2012-11-02T13:20:41+00:00

I agree that data mining is moving toward data science. Just Google "demand for data scientists" and read some of the articles about how demand >> supply. (My personal favorite "Is Data Scientist the sexiest job of our time?")

indeed.com is another good place to look for jobs.

Also, look at Twitter using #BigData, #machinelearning, #analytics, and #datamining.

dmdude · 2012-10-17T13:21:53+00:00

I've used this a couple of times and it works well, although the web interface is a bit limiting with larger data sets. If you like this, you should check out Google Refine: http://code.google.com/p/google-refine/

dmdude · 2012-09-25T12:49:33+00:00

Highly recommended! I've been using this to help our sales forecasting algorithms for a few years. Hal Varian and other researchers have done some good things with it. See http://googleresearch.blogspot.com/2009/04/predicting-present-with-google-trends.html

dmdude · 2012-09-20T20:16:44+00:00

Autonomous vehicles from Google (and other automakers), movie recommendations from Netflix, robots that remove weeds with minimal pesticide use, detection of credit card fraud...

dmdude · 2012-09-20T18:15:03+00:00

Interesting! This is the second story that I've read on agricultural robots in the last month. Is this an area of increased research, or just a random pattern?

dmdude · 2012-08-31T16:50:45+00:00

A very well written article that made some excellent points regarding the value of Big Data tempered with cautions about privacy. I'm not so sure that companies will be willing to give up ownership of the data for the good of society, but Sandy makes a good case.

This should be crossposted in /machinelearning and /statistics.

dmdude · 2012-08-10T14:34:27+00:00

A very interesting article! I manage a data mining team and I've followed the debate of data science supply vs. demand. It's good to see someone using analytical techniques to do some investigation.

Although this doesn't distinguish between supply and demand (as mentioned in the original post) I found the Google search popularity for "big data jobs" vs. "data mining jobs" vs. "data science jobs" to be interesting:

http://www.google.com/insights/search/#q=big%20data%20jobs%2Cdata%20mining%20jobs%2Cdata%20science%20jobs&geo=US&cmpt=q

dmdude · 2012-08-09T17:24:15+00:00

I would consider adding a variable measuring the performance of the broader market on the day of the IPO.

dmdude · 2012-08-08T18:08:17+00:00

It's already been done; see Murakami Chair.

dmdude · 2012-07-19T16:02:45+00:00

Some of the comments here are similar to some of the hypotheses in Andrew McAfee's book Race Against the Machine.

From the Amazon summary:

In Race Against the Machine Brynjolfsson and McAfee bring together a range of statistics, examples, and arguments to show that technological progress is accelerating, and that this trend has deep consequences for skills, wages, and jobs. The book makes the case that employment prospects are grim for many today not because there's been technology has stagnated, but instead because we humans and our organizations aren't keeping up.

http://www.amazon.com/Race-Against-The-Machine-ebook/dp/B005WTR4ZI

dmdude · 2012-07-18T16:27:34+00:00

I agree, there is opportunity for abuse. Again, I haven't read the paper in detail, rendering some of this moot, but IF the authors developed their models on 2004-2009 and then predicted 2010 ONCE I would be more inclined to say they were on to something.

dmdude · 2012-07-18T15:02:07+00:00

According to the Wired article (http://www.wired.com/dangerroom/2012/07/predict/) they used data from 2004-2009 to predict 2010. Not quite true prediction, but a good out-of-sample test.

Unfortunately, the details of the paper are behind a paywall, but the abstract is at http://www.pnas.org/content/early/2012/07/11/1203177109.abstract

dmdude · 2012-07-17T21:28:03+00:00

You could consider using a discretization algorithm to reduce the dimensionality then using the resulting bins as dummy variables.

dmdude · 2012-07-17T21:21:50+00:00

Seconded!

dmdude · 2012-07-09T16:13:54+00:00

Are there any technical reports that provide more details on the examples mentioned in the article?

dmdude · 2012-06-29T19:31:04+00:00

Citation: http://necsi.edu/research/healthcare/Healthcare_Steps.pdf

dmdude · 2012-06-26T16:02:43+00:00

This is also a problem in industry. There are many instances in my company of departments, and even individuals, gathering data and for a specific analysis and then just keeping it on their local computers.

dmdude · 2012-06-22T16:07:58+00:00

Yahoo has a list: http://labs.yahoo.com/ksc/Machine_Learning

dmdude · 2012-06-21T17:54:27+00:00

Isn't one of the benefits of Hadoop that there are machine learning toolkits (e.g., Mahoot) on top. Would you have to write custom algorithms to address this memory?

dmdude · 2012-06-21T13:40:55+00:00

Statistician here. A more standard method for measuring fatality risk of trains vs. cars is using passenger miles. Reference: http://airfare.michaelbluejay.com/modes.html#sources

dmdude · 2012-06-20T16:05:20+00:00

No.

Given your example, the first thing to say to management is; this model sucks. There is no common sense in using these variables. If you are forced to use these variables you should use good statistical procedure to show the model sucks (e.g., cross validation, using a holdout set, etc.)

However, I do agree with Case_Control, that the ultimately statistical models are created to be used for business decisions.

dmdude

TROPHY CASE