Research And Application Of Network Public Opinion Analysis Method Based On Spark

With the rapid development of Internet technology,people use the Internet more and more in their daily life.More and more people are also accustomed to expressing their views about social hot spots and current affairs news on the Internet.But as a free and virtual platform,it doesn't mean that the Internet doesn't need any management,conversely,it is a new responsibility of government staff in the Internet age to find hot topics in time,to curb the spread of false rumors,to guide public opinion in the right direction,and to ensure a good Internet environment.However,it is the first difficult point of the research work that how to extract topics from it,and timely indicate the direction of management and guidance in the future in the face of such a large amount of text data.This thesis combines natural language processing,machine learning and big data processing to solve this difficult problem.This thesis proposes a topic detection method based on the Single-Pass-SOM combination model with multi-feature fusion for topic detection in the analysis of network public opinion.The topic detection method is divided into two parts,one is text representation and the other is topic clustering.For text representation,this thesis puts forward the LDA&&word2vec text representation model based on time decay factor,and the model uses LDA model to extract theme features,and uses word2 vec model to extract semantic features,and combines the two features of text,and designs time decay factor to add time features to the model,so as to obtain more comprehensive text information and improve the accuracy of topic detection.For the topic clustering,this thesis puts forward a single-Pass-SOM combined clustering model,which combines the advantages of Single-Pass clustering algorithm and SOM neural network,and uses Single-Pass clustering algorithm as a rough clustering model,and uses SOM neural network as a fine clustering model for improving the accuracy and recall rate of topic clustering.In this thesis,a series of comparative experiments are designed to verify the validity of the topic detection method.In addition,this thesis nalso applies the Spark Distributed Computing Framework in the topic detection method proposed in this paper,and uses the parallel computing method to improve the efficiency of the algorithm.