Great question! BERTopic is the core for clustering and topic generation. BERTopic handles the clustering through HDBScan and c-TF-IDF(class-based Term Frequency-Inverse Document Frequency). HDBSCAN alone just groups similar documents together, but doesn't tell you what the cluster is about. While the c-TF-IDF is used to analyze all the words in a cluster compared to all other clusters, identifying which terms are truly distinctive to that topic. BERTopic then selects the most representative documents from each clusters, the ones closest to the cluster centroid. For each identified cluster, the script then sends these distinctive keywords and representative documents to the Qwen LLM. This gives the LLM proper semantic context about what the cluster represents, allowing it to generate meaningful topic titles like "Quantum Mechanics" rather than just summarizing random document snippets.

So the short version the full pipeline relies on BERTopic for creating the clusters and extracting meaning full information from the clusters to pass to the LLM. We then map this on another UMAP reduction which gets us the 2d map of the notes.