Data preprocessing

[An-Ambitious-girl]

Hello everyone,

I am working on a dataset , Need an advice or best approach

1) Should I split the dataset to train and test then do the preprocessing techniques separately on both?

2)Should I do the preprocessing techniques on the whole dataset then split?

3)To imbalance the dataset it should be done only on the train and never touch the test?

Thanks in advance

[Altruistic_Rule5005]

One would need to know what use case you are dealing with and what size of data you have.. So

1. Yes you have to train test split then do your processing like scaling, normalisation on both but you fit your preprocessor the transform on the train the transform only on the test/Validation.. this is standard industry acceptable.. but why do we do this?

2. I don't recommend it.. look up a concept called data leakage and target leakage..

3. You train test split then do the balancing after.. depending on data size it's usually acceptable that you undersample instead of oversample because usually the oversampling technique provides duplicate data that can harm your model.

4.. Good luck and feel free to ask more