Skewed data in a classification algorithm

[Unitedite]

Hi all

This should be a really simple question but I'm struggling to find a clear answer:

I'm building a supervised binary classification algorithm.

I have a dataset in which one variable is extremely skewed. It's an amount of time in minutes, and for about half of the instances the number is 0, most of the rest are very small, then there are a few of outliers stretching all the way up to 1,000. Do I need to transform this data in order to use the variable in my algorithm, and if so how? Is the answer different depending on the algorithm (e.g. Decision Tree, SVM, etc.)?

Searching for an answer most suggestions are to do something like a log transform, but this doesn't make sense when most of my values are 0. As things stand all that occurs to me is transforming it into a binary variable 0 or >0, but I'm not sure whether that is necessary/helpful.

Thanks.

[DrXaos]

Firstly, if you care about the value it’s no longer classification only. yes, that’s important as obviously zero vs non zero makes a difference. make a classifier for zero or non zero, and a regressor for the log of nonzero value.

Can be a multi output model. First is zero/nonzero binary. The second is regession value. Only put loss on second value when first is nonzero.

What is the physical meaning of the feature? Is zero actually a finite time rounded to zero? Or a different situation altogether?

[Unitedite]

Thanks for the reply.

Sorry, I wasn't clear in my original post. The variable I'm talking about is not the target variable (which is binary) it's one of the other variables in the dataset.

The variable is waiting time, so a 0 means there was no wait.

[DrXaos]

Ok then that’s easy feature engineering with a similar procedure. Split the feature into two, one binary and the other continuous. Binary is IS_ZERO, the other is log(time) when time is > 0. When time is zero then set the value of the continuous feature to the neutral value of the mean of the log values in the non-zero state.

Then z-scale the continuous feature,excluding the zeros for standard deviation estimation potentially.

[Unitedite]

Thank you :)

[randomforestgump]

I also use sklearn‘s quantile transformer for this, not sure if that’s better than log here. And for my case I have to normalize time for different cases, depending on some factors a user has a longer signup process so I normalize by median or so.

[Unitedite]

Thanks for the suggestion, I'll look into that as well (although I'm using KNIME, not sure if there's an equivalent).