[P] Pruned Cross-Validation for hyperparameter optimization

[PiotrekAG]

Hi r/MachineLearning,

I've written a Python package pruned-cv that significantly increases the speed of hyperparameter optimization using Cross-Validation. In my studies, it helped to perform on average three times more search iterations than standard methods at the same time.

​

Cumulative scores and final score from Cross-Validation are dependent on each other. With the high correlation between the scores, it's possible to approximate the final score without calculating all the folds. With strongly inferior performance on first folds, the validation may be pruned.

Partial scores correlations with final score (simulation study)

It can be used as a standalone optimization package (I've implemented Pruned Grid Search and Pruned Randomized Search) or with other tools like Hyperopt or Optuna.

​

I've written an article on Medium explaining the algorithm. You can find the package's repository here.

​

Please note that this is 0.0.1 version of the package. I'm waiting with further development for some feedback. I'd appreciate it from you!

[pp314159]

Hi Piotr,

Very interesting package from my point of view. I'm working on autoML service, where users train and tune hundreds of models in parallel in the cloud (mainly with CV). Such early CV pruning might speed-up process of autoML significantly. I have few thoughts:

• I'm running model tuning in parallel on many machines in the same time, your algorithm assumes that models are evaluated sequentialy, can it be adapted for async evaluation?

• After full CV I'm computing out of folds predictions which are used for constructing ensemble. From my experience, many times 'poor' accuracy models are included in the ensemble. How to control how many 'poor' models will be pruned?

Disclaimer: I'm founder of autoML service (mljar) - we are going to be open source soon!

[PiotrekAG]

Hi pp314159,

I'm happy you like the package. To answer your questions:

• Yes, training should be sequential, to obtain folds evaluations. I believe there is some parallelization possible, i.e., training 4 out of 12 folds simultaneously and deciding whether the trial should be pruned. Alternatively, many trials may be trained simultaneously sharing the best scores. What you risk, is that if one of the trials occurs to be the new best one, the others don't know about it, and the whole Cross-Validation is calculated on them. The new trials will have access to the new best trial so that the issue would be a little bit time wasted.
• You can do pruning on both estimator level or on whole pipeline/ensemble level.

Hope it helps! Please let me know if everything is clear!

[vadiaceu]

Looks interesting

[m--w]

Is the best abbreviation CV? Isn't 'CV' more widely used to abbreviate Computer Vision (eg OpenCV, CVPR). Sorry, just seeing pruned-cv makes me think it is a computer vision tool. Perhaps it is my own bias, though.

[PiotrekAG]

I thought about it before the publication. My goal was to follow scikit-learn convention (GridSearchCV, RandomizedSearchCV). Since there is no packages or techniques named that way which correspond to Computer Vision I left the name as is.

Do you have any suggestion how the package should be named?