| With the rapid development of mobile Internet,Search engines,Blogs,WeChat and other applications make people's lives and work easier and more efficient,and the vast amount of short text data will be generated in these application every day.So it is of great guiding significance to extract useful information from these data.Short text clustering can process a lot of data effectively and it can mine the relationship between short texts by clustering analysis.Moreover,short text clustering has a broad prospects,which is the basis for information retrieval,user portraits,personality recommendations and other research fields.However,short text has the characteristics of lack of words,sparse features and many informal statements,and traditional text clustering algorithms perform poorly on it.Many studies have shown that Word Embedding is characterized by dense features and ease of use,and applying it to short text clustering will imporve the clustering effect.This paper covers the following content:Short text clustering based on Word Embedding and EWMA(Exponentially Weighted Moving Average).Using Word Embedding for text representation can get it's semantic features,but for short text,it has fewer words and the impact of each word is crucial.Influenced by those "abnormal words",averaging the word vectors of all words in the text to represent the text often has poor performance.In order to solve this problem,this paper combines Word Embedding and EWMA for text representation.And experiments show that the clustering effect has been effectively improved.Short text clustering based on Word Embedding and EWMA(Exponentially Weighted Moving Average).Using Word Embedding for text representation can get it's semantic features,but for short text,it has fewer words and the impact of each word is crucial.Influenced by those "abnormal words",averaging the word vectors of all words in the text to represent the text often has poor performance.In order to solve this problem,this paper combines Word Embedding and EWMA for text representation.And experiments show that the clustering effect has been effectively improved.Short text clustering based on Word Embedding and improved WMD(Word Mover's Distancence).Using the calculation result of the word vectors to represent the text,it is inevitable that some semantic information will be lost.It is proved that using WMD distance formula to calculate the similarity between texts can improve the similarity measurement effect.However,the formula is computationallycumbersome and inefficient.Aiming at this problem,this paper takes into account the characteristics of short text and simplifies the WMD distance formula.After that,a density-based short text clustering algorithm is proposed,and the algorithm introduces the technique of "inverted index".It can reduce the time complexity of the clustering algorithm and improve the efficiency of clustering.An application example of short text feature mining based on clustering.In order to verify the effect of the above algorithm in reality,this paper constructe a binary classification task from the Weibo recommendation system.After clustering the Weibo content,this paper takes the distance from the text to each cluster as a text feature.Experimental results show that the method proposed in this pape are obviously better than before,which shows the value of this dissertation. |
