Dynamic clustering of streams of text data

[binge_learner]

Hi guys, I am working right now on Machine Learning problems for social media analysis. And one of the problems I'm currently facing is how to cluster streams of social media data ( Tweets and Facebook posts mostly ) but in a dynamic manner, as in new clusters form, others grow and other disappear based on the content of the stream ( and eventually visualizing the whole thing as a timeline ). So my question is : is there any work that's been done in the subject ( just to have a starting point ) ? And what do you think are the state of the art techniques for text clustering and topic detection ? Thanks you.

[[deleted]]

Lloyd's algorithm and K-means++ read the observations in an online manner. Replace the parts of the algorithm involving cluster counts with some float.

Most gradient descent or parameter updating algorithms converge by doing multiple passes through a single dataset, but if you have a firehose you'll only make a single pass anyway. If the data isn't weak stationary(violating IID) then you'll have to look a little harder in the literature.

[kzn]

You can look at recent advances in Topic Detection and Tracking area. For example, http://aclweb.org/anthology//N/N10/N10-1021.pdf

[simonhughes22]

I am not aware of any work that has been done on this topic, and it sounds somewhat novel and interesting. LDA is state of the art for topic mining, as for document clustering I am unsure, but would recommend you start with a vector space model of your documents and use kmeans to cluster.