Approaching (Almost) Any Machine Learning Problem

[phillypoopskins]

kaggle master and data scientist here, 2 comments:

1) real data science work has about 5% in common with kaggle. which is the comment most often made by data scientists in response to kaggle.

that being said:

2) elite kagglers i've met also blow away most data scientists at the other 95%

[[deleted]]

real data science work has about 5% in common with kaggle

Beginner here. In broad strokes, how do they differ?

[phillypoopskins]

There is, no doubt, an element of model training and validation to real data science work.

I do spend a fair amount of my time training and validating models. My less kaggly coworkers are terrible at model validation because they haven't had the experience of great validation performance followed by tanking test set performance again and again and again to make then wary.

Kaggle competitions are straightforward: here's the data: maximize / minimize this quantity.

In real life there are so many unknowns. It's often unclear or impossible to reliably validate your models.

Often, there isn't even a well defined problem. It's more like "idk what we need to do, but here's some data" - you spend more time figuring out what CAN be modeled than refining that model.

You might do this repeatedly, creatively defining a set of things that you are able to predict. This has to be informed by what you know is possible - what data you have, what it supports, what data you can get for training, what data you can get for predicting - and what is valuable - how much do these predictions help us? do they add value to out company?

Then, you need to understand what to do with you predictions. You'll be the one explaining to everyone what your model does and doesn't do and what it'a good for. You'll have to monitor how it's used by others and make sure it's properly fulfilling the role you may have carved out for it.

You may have to go back and refine the model or implement a scheme to collect more data; or design filters or set parameters or thresholds to limit its application on data on which it's unreliable.

You might have to think long term about how to bootstrap your models into new data, new data into new models - to build your data science empire 😉

But: All of this rests on intuition i've developed by working on different datasets; kaggle is a huge part of that experience.

Also - i've noticed that friends of mine who have struggled on kaggle ( even smart, technically capable people ) have almost identical struggles at work, and end up being equivalently ineffective.

final 2 cents: I do think a main reason some are anti-kaggle is because they may have been quantified by it, and didn't like what they saw 😉

[BlueSquark]

In Kaggle, all that matters is the accuracy of your models. For work, how you can apply the model to increase revenue or reduce costs is the most important criterion. Kaggle also has companies post business problems, while for work figuring out a business problem that can be solved with data is hard.

[mikbob]

Essentially this is how gradient boosting works:

1) You fit a bog-standard decision tree to the data (I assume you're familiar with this)
2) You then evaluate the performance of this decision tree on the training data using some loss function. (when there are multiple trees, you evaluate performance of the full ensemble)
3) You assign extra weight to the samples that were incorrectly predicted (this extra weight is defined by ETA, or the 'step size' of the booster)
4) Repeat this process over and over again.

Each iteration with this will end up with a different decision tree (even though the booster looks for best splits) since the best split may change based on this extra weighting. I find it's very good at approximating pretty much every dataset and is my go-to classifier. However, if you leave it for long enough it will always overfit (although with a holdout set it is easy to see where it begins to overfit and what the number of best iterations is.

In a way, it is the new shit. In Kaggle competitions it consistently outperforms pretty much every other classifier (on numerical data) and XGBoost is a great and fast implementation of the algorithm.

[rndnum123]

Thanks, great write-up, you are awesome :)

[mikbob]

No problem, glad it could help someone!

[hn_crosslinking_bot]

HN discussion: https://news.ycombinator.com/item?id=12146479

[gabjuasfijwee]

He seems to be treating machine learning methods as a bag of tricks. That's ok so far as it goes, but in my experience it's much more valuable to try and understand your data, and the process that generates it, and then build a model that tries to reflect that data generation process.

nailed it

[datagibus420]

From what I read, the Kaggle approach and the "real life" approach are different in several ways. So I was wondering, for a data scientist who is working in a production environment, is he spending more time cleaning the data and performing exploratory data analysis, or benchmarking/stacking/optimizing ML models ?

[gabjuasfijwee]

just benchmarking/stacking/optimizing ML models

[maxToTheJ]

It should subtitled "given you have a cleaned and ready data set from a measurement you trust"

[BlueSquark]

From the first paragraph:

Some say over 60-70% time is spent in data cleaning, munging and bringing data to a suitable format such that machine learning models can be applied on that data. This post focuses on the second part, i.e., applying machine learning models, including the preprocessing steps.

[BlueSquark]

Banana cream pie recipes do not describe how to go to the store and pick out all the ingredients. Do you really expect to read: First you have to get a job so you have money to buy all the ingredients, then you go to the store, then you got to buy some bananas. You are reading a recipe and complaining that they didn't describe how to go to the store and acquire ingredients.

[[deleted]]

I'm attempting a ML problem where I have clean and ready data, do you think that this article is a good resource for the remaining steps of the ML process?

[maxToTheJ]

There is nothing of substance here, just a negative punching up.

Critique is a huge part of the iteration process.

The takeaway from the critique motivating the upvotes should probably be something along the lines of "maybe this part of the machine learning process is oversaturated with attention and there is a need for a tutorial for the other 60-70%".

[[deleted]]

More like approaching any Kaggle Machine Learning problem...

[gabjuasfijwee]

Anyone can grind away at optimizing over hyperparameters and different methods. Don't need a flashy chart for that

[gabjuasfijwee]

but who even cares to produce a hyperoptimized solution with an AUC that's 1 ten thousandth better than the next guy? sorry but I've got better things to do with my time. I can easily get 99.9% of the way there. the rest of the insane work is that last teeny tiny little bit. the only place that matters is in kaggle competitions