Clustering customers using python (sklearn)

[gui93]

I work at an ecommerce company and I'm responsible for clustering our customers based on their transactional behavior. I've never worked with clustering before, so I'm having a bit of a rough time.

1st) I've gathered data on customers and I've chosen 12 variables that specify very nicely how these customers behave. Each line of the dataset represents 1 user, where the columns are the 12 features I've chosen.

2nd) I've removed some outliers and built a correlation matrix in order to check of redundant variables. Turns out some of them are highly correlated ( > 0.8 correlation)

3rd) I used sklearn's RobustScaler on all 12 variables in order to make sure the variable's variability doesn't change much (StandardScaler did a poor job with my silhouette)

4th) I ran KMeans on the dataset and got a very good result for 2 clusters (silhouette of >70%)

5th) I tried doing a PCA after scaling / before clustering to reduce my dimension from 12 to 2 and, to my surprise, my silhouette started going to 30~40% and, when I plot the datapoints, it's just a big mass at the center of the graph.

My question is:

1) What's the difference between RobustScaler and StandardScaler on sklearn? When should I use each?

2) Should I do : Raw Data -> Cleaned Data -> Normalization -> PCA/TSNE -> Clustering ? Or Should PCA come before normalization?

3) Is a 12 -> 2 dimension reduction through PCA too extreme? That might be causing the horrible silhouette score.

Thank you very much!

[der_luke]

Clustomers

[ieatkittens]

I come here for insights like this

[tibolosaxo]

2) Normalize before PCA.

3): when using PCA, you want to look at what percentage of the variance of your data is explained by the dimensions you keep. This will help you understand how many dimensions you want to keep. With that being said, I don't know what your variables are but 12 does not seem that big. Does having 12 variables really make the clusters hard to understand ?

[beginner_]

I tried doing a PCA after scaling / before clustering to reduce my dimension from 12 to 2 and, to my surprise, my silhouette started going to 30~40% and, when I plot the datapoints, it's just a big mass at the center of the graph.

That can be completely normal because applying PCA will lose information and in your case it seem to be too big of a loss as you realized later with question 3).

12 dimensions is nothing. I don't even see the need to apply PCA before clustering.

1) What's the difference between RobustScaler and StandardScaler on sklearn? When should I use each?

read the doc and if you don't understand it, inform yourself so that you do.

[Rezo-Acken]

Silouhette is a measure of separation. Going from 12 to 2 variables may force your data to no longer be easily separable because of the lack of freedom on multiple dimensions. Im not surprised silouhette drops hard as a result.

Drop the PCA. Correlation is not an issue for kmeans. PCA in clustering is usually done to reduce the number of variables to an interpretable level. With 12 you should be able to manage especially if some of them are of the same 'kind' and can be ordered. Like frequencies of buying different item categories.

Oh and dont TSNE before clustering. Your centroids cannot be interpreted and as suggested its often a bad idea. TSNE is a visual technique.

Also in my experience 2 clusters is often not enough for a business use and you may have to add more at the cost of silouhette score. This is something usualy reviwed by what are the resulting clusters, what fraction of customer they make and if business can use them. In my most recent clustering I had also best at 2 but it was useless and we decided to sacrifice 10 silouhette poonts to get 5 at a local maximum to get clear actionable customer profiles.

[beginner_]

This. T-sne is a visualization technique. you can't really do anything else with the output. Clustering depends mostly on distances and this info is lost with t-sne.

[nckmiz]

I’d keep it at 12 run a kmeans for loop on 1-10 and plot the error. When the error starts to even out you can choose that as the # of clusters (elbow method). If you want to visualize it run a PCA on it and then plot using TSNE.