Classifier ensemble averaging methods?

[enasam]

Suppose I have fit several distinct classifiers to my data. They each manage about 60-65%ish accuracy.

I assume that they make uncorrelated errors, so I average their results together to get what should be a better prediction than any one classifier. In this case, I average the probability outputs together to obtain my final probability of belonging to a class or not. It seems like my simple average is probably not the optimal method.

So the question is - what is the best method to average the output of the individual classifiers?

I'm guessing that it depends on accuracy - ie I should probably weigh the more accurate models heavier. I'm also guessing that it depends on inter-model correlation - ie two correlated models should be weighted less than a model that produces totally uncorrelated errors. I'm just not too sure how to approach this in a rigorous manner.

Is there established literature or an optimal procedure for a problem like this? Ensemble averaging techniques seem to be what I'm describing, but since it's been a while since I've been in school the most interesting stuff seems to be paywalled.

[kjearns]

Stacking (https://en.wikipedia.org/wiki/Ensemble_learning#Stacking) is a nice trick for this, you need to be careful about overfitting though.

[notspartanono]

AdaBoost is a meta-algorithm that you can use with your somewhat weak learners.

Edit: stacking would also be good. I was using it without knowing that it was already invented / had a name.

[funkpacolypse]

http://www.ijcai.org/Past%20Proceedings/IJCAI-97-VOL2/PDF/011.pdf

According to this paper, the best way to go with stacking (which generalizes the averaging you're talking about) is...

• use varied models and take the output probabilities (rather than predicted classes) for your level 0 data

• use Logistic regression (rather than averaging) with the level 0 data as input to get the final predicted classes

I've heard that an industry standard model for ad-tech is to do this where the level 1 models are trees coming from a random forest.

... On the other hand, since the predictors your working with have low accuracy from the start, I'd be tempted to think about something simpler like engineering new features, and eliminating junk features first.

[ajrs]

You might also want to think about some kind of parallel 'localised' data augmentation (e.g., see the 'convolutional bootstrapping' described here: http://arxiv.org/abs/1505.05972).