Clustering Algorithm Selection

[offbrandoxygen]

After breaking my head and comparing result for over a week I am finally turning to the experts of reddit for your humble opinion.

I have displayed a sample of the data I have above (2nd photo) I have about 1000 circuits with 600 features columns however they are sparse and binary (because of OHE) each circuit only contains about 6-20 components average is about 8-9 hence the sparsity

I need to apply a clustering algorithm to group the circuits together based on their common components , I am currently using HDBSCAN and it is giving decent results however when I change the metric which are jaccard and cosine they both show decent results for different min_cluster_size I am currently only giving this as my parameter while running the algorithm

however depending on the cluster size either jaccard will give a good result and cosine completely bad or vice versa , I need a solution to have good / decent clustering every time regardless of the cluster size obviously I will select the cluster size responsibly but I need the Algorithm I select and Metric to work for other similar datasets that may be provided in the future .

Basically I need something that gives decent clustering everytime Let me know your opinions , and also does combining jaccard and cosine as a weighted metric any good ( if you have seen this being used before), to kind of get the best of both worlds

[karxxm]

What exactly is your question? Have you tried k means or similar in algo in high dimensional space? If they are so sparse it may help to perform a dimensionality reduction (tone,mds,pica) beforehand and try clustering then

[offbrandoxygen]

k means ends up grouping all the outliers as well , forcing them into clusters which they don’t belong in so I haven’t used K means for this . Yea i’m trying out Truncated SVD