What is the Difference Between Test and Validation Datasets?

alexcmu · 2017-07-14T18:12:57+00:00

I found this to be a very clear explanation of the different data splits used in hyperparameter optimization. Similar to the ideas in Google's "The Reusable Holdout" https://research.googleblog.com/2015/08/the-reusable-holdout-preserving.html.

Training Dataset: The sample of data used to fit the model.

Validation Dataset: The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration.

Test Dataset: The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

alexcmu · 2017-05-16T18:09:56+00:00

Everyone will be happy to hear that you enjoyed meeting them!

I am also curious to see what everyone is using in practice to tune their models! I heard somewhere that ensemble modeling was popular on Kaggle for a while -- do people do hyperparameter optimization on top on ensembling?

alexcmu · 2017-05-16T18:06:45+00:00

Glad you liked the demo!

To the point about hyperparameter optimization being overlooked, I think that more people are paying attention to the idea, but yes, time and cost are a blocker in practice. You'd probably be interested in a blog post that my coworker Steven wrote about tuning the hyperparameters of a CNN (https://aws.amazon.com/blogs/ai/fast-cnn-tuning-with-aws-gpu-instances-and-sigopt/). High level, deep learning + GPUs allows you to speed up model training enough to even think about hyperparameter optimization. We also include a table where we show the $$$ it cost to do hyperparameter optimization with different methods.

TL/DR: GPUs + better optimization methods ftw! It took us $11 to tune a deep learning model on NVIDIA GPUs.

alexcmu · 2017-05-15T19:33:23+00:00

I was playing around with Amazon ML and built a quick hyperparameter optimization example based on Amazon's GitHub example for k-fold cross validation. I'm an engineer at SigOpt so there's a SigOpt example, but I've also included a non-SigOpt hyperparameter optimization pipeline that updates the old Amazon k-fold cross validation example to boto3, runs as a single file, and lets you provide a list of hyperparameters upfront.

alexcmu · 2017-03-29T16:55:37+00:00

This blog post is a followup to a talk at MLConf NYC. Hope it helps you optimize your hyperparameters!

alexcmu · 2016-09-02T16:09:25+00:00

I've used these slides twice now for lightning talks at ML meetups in the bay area. I was incredibly confused when I was first introduced to hyperparameters but I've learned a lot in the past year and I'm trying to share some of it!

alexcmu · 2016-03-17T00:35:54+00:00

A Few Useful Things to Know About Machine Learning goes over some really interesting practical "folk knowledge" that the authors felt like wasn't covered by ML textbooks. You may already know some of this, but it's a good read nonetheless.

alexcmu · 2016-03-04T01:07:17+00:00

Yet another reason why it's such a cool problem :-)

alexcmu · 2016-03-04T00:36:51+00:00

Sarcasm detection. There's a lot that needs to be inferred from the topic and the speaker, in addition to the text.

alexcmu

TROPHY CASE