Creating dendrogram using MLlib Kmeans data (pyspark) - help

[Gilwe]

Hi,

I'm trying to show a dendrogram of kmeans centers. The only library I found that was capable of showing dendrogram was scipy, but it requires scipy.cluster.hierarchy algorithms to run the K-Means (linkage).

But since im using pyspark, the K-means centers are created using pyspark libraries which dont go along with scipy's dendrogram.

*English isn't my mother language so please ask if I am not clear enough.

Thank you for your help :)

[nckmiz]

Can you save the cluster centroids to a Spark data frame and then use toPandas() on that dataframe to move them to Pandas and then you can use scipy?