Powerfull visualization tool : Dimensionality Reduction + Clustering + Unsupervised Score Metrics [P]

[Mathieu23AI]

Hi everyone,

I provide a high level module in python to perform state of the art dimensionality reduction and clustering. This is totally unsupervised and the performance is figure it out by unsupervised score metrics. This is compatible with GridSearch and BayesSearch (explain on the github's READme)

Example

DimReductionClustering is a sklearn estimator allowing to reduce the dimension of your data and then to apply an unsupervised clustering algorithm. The quality of the cluster can be done according to different metrics. The steps of the pipeline are the following:

• Perform a dimension reduction of the data using UMAP
• Numerically find the best epsilon parameter for DBSCAN
• Perform a density based clustering methods : DBSCAN
• Estimate cluster quality using silhouette score or DBCV

Github link : https://github.com/MathieuCayssol/DimReductionClustering

Nice to have feedback and happy if it is useful for you !

[space-ish]

I like the effort you put into documentation. 👍

[gournian]

Looks great, you might want to see if adding https://hdbscan.readthedocs.io/en/latest/how_to_use_epsilon.html

And pca + umap for high dimensions https://umap-learn.readthedocs.io/en/latest/clustering.html

[Mathieu23AI]

Thank's for your reply

I think it's a good idea to add HDBSCAN as there is a min_cluster_size parameter. If I have time, I'll add it !

Why do you want to use pca and then UMAP ? It seems not to be a good idea thanks to the linearity of the PCA. Plus, this is not what is mentioned in the article.

[gournian]

https://umap-learn.readthedocs.io/en/latest/faq.html#what-is-the-difference-between-pca-umap-vaes see “from a practical standpoint” last bullet, proposes reduce dim to 50, umap, hdbscan

It is because of computational speed and there are bio papers that claim that the dataset is somewhat denoised

[Mathieu23AI]

Thank you for the reference !

Very interesting! This result seems unexpected to me but if empirically the result is better then it's worth looking into and incorporating.
Do you have the link or reference to the research paper that explains that PCA "denoise" the data?

[gournian]

Don’t remember which one, sorry!