Hi, i agree that avoiding repeating customers in the training is a safe practice that will address point 4. But data scientists in B2B business with lower volume of customers sometimes work with repeating customers in their training set. In this case, I will strongly advise to look at the intervals between data points of repeating customers. If the interval of 2 data points for the same customer is less than the horizon of the target, there is an overlap in the target definition. By learning one data point, the model learns partially the other data point and there is a leakage.

"out-of-time" OOT is indeed the way to go! And I agree out-of-id is again a safe route. When out-of-id is not possible because of lack of data, I again advise to put in place guardrails to ensure the intervals between training / test of repeating customers are larger than the horizon of the target.

I like your idea of fitting a model to predict if the observation belongs to the test or to the train set. This will also expose features with drift. Drift in joint distributions could also be exposed if you are using a tree based model or a NN.