| As new words appear more and more frequently on online platforms,automatic recognition of new words by technical means is an important part of public opinion analysis,which directly affects the quality of Chinese word separation and indirectly affects the effect of text mining.When current Chinese word separation tools cannot accurately identify new words,they often cut the word strings incorrectly and generate a large number of "word fragments",resulting in partial loss of information in the complete word structure.Based on the above background,this paper starts the subsequent research based on the summary and reference of existing literature.After sorting out the advantages,disadvantages,principles and effects of existing Chinese new word discovery technology routes,a statistical and rule-based approach is selected for new word discovery research,and the idea is elaborated from the conceptual level.After that,the framework of Chinese new word discovery is designed with public opinion analysis as the main application scenario,including two major processes: building candidate word sets and new word detection.First,in the part of constructing the candidate word set,in order to retain the original information of the corpus as much as possible,the N-gram algorithm is used to extract repetitive strings,and the strings are filtered by combining word frequency,point mutual information,left and right adjacency entropy and other word feature statistics.The training corpus set provided by Peking University in the Second International Chinese Word Sorting Competition is used as the experimental data and combined with the provided control corpus to validate the algorithm process.The experiments demonstrate that the optimal threshold of feature statistics is determined by the grid search method in the presence of the standard word separation control,which can effectively filter the junk word strings and improve the new word recognition.Secondly,in the new word detection part,in order to solve the problem that the determination of the threshold value of feature statistics needs to be controlled against the corpus,it is proposed to supplement the algorithm by using SVM classification model and Jieba subscripts.The focus is on integrating the SVM classification algorithm into the Chinese new word discovery process by matching the annotated training set,and updating the existing word separation lexicon by integrating the results obtained from the support vector machine into the lexicon,so as to improve the existing word separation tools.The results show that the combination of word features and SVM can achieve the recognition and extraction of new words and enhance the new word recognition ability,and the radial basis kernel function new word recognition preciseness is 74.8% and the recall rate is 79.3%,which is the optimal experimental result obtained.From the perspective of research continuity and practicality,the method proposed in this paper is used for opinion analysis.We crawl microblogging blog post data with new energy vehicles as the subject word,and add the identified new words to the Jieba dictionary to improve the word separation accuracy.And the public opinion is analyzed by word cloud map,LDA topic model and other techniques.The results show that people are generally concerned about the development prospect of new energy vehicles,and the overall orientation of online public opinion is positive. |
