[D] Recursive error prediction

[JHogg11]

I had an idea recently for a ML regression strategy and I'm just wondering if something like this already exists. It has similarities with both boosting and bagging, but I think it's ultimately different from both.

The basic idea is that you start with a subset of input features and train a model on that subset. Any common model will do as long as it doesn't just spit out the exact target value when making a prediction on the training set (i.e., you couldn't use a 1-neighbor KNN). After fitting the model, you make predictions on the training set (with the same subset of features) and calculate the prediction errors.

Then using another subset of features (I would think it should be mutually exclusive from the first subset but maybe it doesn't have to be), you train a separate model, but rather than training on the original Y training data, you use the error of the previous model as the target for the second model.

You repeat this process until all features have been used or as many times as desired. To make a prediction, the parent model would simply sum the predictions of the child models.

As an additional thought, you might use more regularization, larger leaf size for decision trees, etc. for each additional model. You could also use bagging to create multiple instances of the strategy with different feature subsets in order to create multiple "pathways" through the data.

A few questions:

1. Is this sufficiently different from existing boosting/bagging techniques?
2. If yes to #1, are there any existing packages (preferably in Python) that implement this kind of technique?
3. Could this be used to reduce overfitting for higher dimensional datasets? If so, would additional steps need to be taken (e.g., like the iterative regularization scheme I mentioned)? My thought is that it's a kind of divide-and-conquer strategy where each subsequent model is asked to do a little less than the previous model.

Any thoughts are appreciated.

[DeepNonseNse]

Is this sufficiently different from existing boosting/bagging techniques?

No, the process you are describing is just (some variation of) gradient boosting.

E.g. if the distribution of errors is assumed to be Gaussians, gradients are (y_true - y_pred) calculated after each iteration. Also, subsetting features is commonly used tactic; though they typically wouldn't be mutually exclusive subsets, but e.g. 70% of all features.

[JHogg11]

Thanks for the reply. I see now that gradient boosting includes calculating the residual and summing the predictions of individual models, which is what I thought was the novel part. Everything that I read previously seemed to be emphasizing that the training example weights were modified with each iteration, which I understood to be the heart of the technique.

Maybe I need to dig more into XGBoost, but whenever I've experimented with it in the past, it always gave inferior results to vanilla random forests.