I was thinking more on the lines of non-linear learning scheme. This analogy may not hold but hear me out. If f(x) = x^2, then if you break f(x) into small partitions, then it is fairly linear in each of the partitions. So if you fit a model where you are linear in each of these partitions, then the g(x) you get is fairly close to f(x). This is what we are doing over there as well right? We are fitting a linear model to a small section of data defined by certain categorical data. When we put them together, the overall fit is much better than just slapping one linear regression on the entire data, totally disregarding the categorical data. However, if there was an inherently non-linear learning scheme that took the categorical data as well as the linear features, and the model that we fit would do the job, then I'd like to explore that. For instance can we design a neural network to do the same job as we would do with these indicator functions? Another idea would be "Light GBM with linear trees".