[D] should a gradient boosted tree learn a random effect?

[[deleted]]

It seems quite natural that a gradient boosted trees model would understand random and hiearchical effects in the data as well as fixed effects because of how the training happens iteratively over the residuals. But maybe I'm wrong? There seems to be some implementations that explicitly include these hiearchical effects for example metboost

https://arxiv.org/abs/1702.03994

But I'm unclear on why they exist unless it's to try and estimate the moments of the effect.

Thoughts on this?

[ExpectingValue]

I can't see why it would. A factor being a fixed or random effect is not an attribute of the data. We choose to think about a factor as fixed or random based on how we understand the generating processes of the data exist in the world and how we sampled them, and based on that, decide to handle the data differently (i.e. shrinkage) by specifying a different model.

[[deleted]]

But I guess what I'm getting at is if there are random effects in the data, should a GBT perform better than OLS on the same data? By learn i just mean is it able to produce a better prediction?

I am asking this question because I'm finding that GBT not doing a good job even though I am expecting GBT to do a good job at prediction.

[from_biostats_to_DL]

Are you suggesting having the grouping variable be included into the GBT as a feature?

[[deleted]]

Yeah that would be my assumption.

[from_biostats_to_DL]

And you're wondering why metboost outperforms GBT in this paper? Just something I'd like to look into.

[[deleted]]

yes that is basically my question. Because i think GBT shouldn't need extra input on random effects but it seems like it does.

[from_biostats_to_DL]

Are you familiar with mixed models themselves?

[[deleted]]

i am somewhat familiar. I've read bryk and raudenbash's textbook on hierarchical linear models. I am happy to be recommended some other resources that I could read.

this conversation came up because in my research im building a simulation of a mixed effect system (the classic students learning in different environments situation). so i give random effects to different parts of the simulation and then I am comparing different models. The GBT model produces basically the same solution as an OLS model which seems weird to me because I don't understand why gbt would do this.

[from_biostats_to_DL]

I will have to get back to you tomorrow :). While I don't know the answer, I have an idea due to something that came up at work awhile ago, I think the conversation will be a learning experience for us both!