Research On Network News Hot Topic Detection Based On LDA Model And Clustering

With the rapid development of Internet technology,the major news portal websites have emerged,and become the main channel for news media and people to obtain information.However,it's difficult for people to access the key information rapidly and accurately due to the complexity,redundancy and rapidity of the network.Besides,it is not conducive to the monitoring of network public opinion.Therefore,the hot topic discovery has been a research hotspot.This thesis researches the traditional topic detection method,according to the deficiency of VSM on Semantic Mining and the insufficiency of LDA on preprocessing,this thesis combines Chameleon Clustering algorithm and applies it on topic discovery,proposes a hot topic discovery method named PLDA-CHC-SP which based on the time window.The main work includes:1)For the weakness of traditional data preprocessing method while using LDA for modeling,this thesis proposes a modeling method which uses the TF-IDF value to filter the words for LDA.It's helpful for enhancing the speed and accuracy of hot topic discovery.2)Many traditional clustering algorithms for word model are based on VSM.This thesis takes full advantage of LDA and Chameleon Hierarchical Clustering,proposes a new method PLDA-CHC(Promoted Latent Dirichlet Allocation with Chameleon Hierarchical Clustering)which combines PLDA and Chameleon Hierarchical Clustering to enhance the discovery accuracy of daily topics.Besides,this thesis proposes a new formula to measure the daily topic heat.3)For the results of daily topics discovery by PLDA-CHC,this thesis uses the Single-Pass incremental clustering algorithm to find a larger time interval topic.Finally,through analyzing the heat of the hot topics,this thesis draws the topic heat development trend chart.The experimental results show that the method proposed by this thesis can improve the accuracy of topic detection and have a certain practical value.