| Chinese Internet has entered the era of the internet and car networking era.Intelligent services are everywhere in our life.A large amount of data information has been produced in different fields and industries.At the same time,text mining technology and processing technology also makes the development of the industry faster and more efficient.The large amount of information left by the users has a high value,but at the same time,the complex network words bring great challenges to the processing and analysis of the text.Therefore,how to automatically identify the meaning of words from a large number of unstructured documents,to calculate the similarity of words,and to develop high-level semantic information and to express it has become the current urgent need.Firstly,the statistical method of word similarity calculation is relatively simple and the result is not accurate.This paper presents a method for calculating the similarity of words based on WEB text.Based on web crawler HERITRIX to grab web pages in the Internet,the HTML format of the data was grabed.After the label information in the pretreatment of web pages and stop words were filtered.Based on ICTCLAS system of Chinese academy of sciences,the plain text information was segmented according to the documents,paragraphs,and sentences as a text unit.The similarity degree of any two things was calculated from the Angle of information theory.Based on this theory,using an index of the text in three paths was builded by Lucene,the cooccurrence probability of two words with middle word individually that appear in the same document,the same paragraph,the same sentence at the same time was calculated.The cooccurrence similarity of words with the middle word in each unit was calculated.And the weighted sum of the similarity of words was calculated,the composite similarity of two words was calculated in three different units.Example results show that guarantee the word similarity computation accuracy,meanwhile simplify the calculation model.Convert the calculation method of the original level of depth to calculate two words in the same language the cooccurrence probability and then calculate the similarity by using the difference of the coconcurrency frequency.This method can make full use of the relationship between the words in the big data text and simplify the calculation model,and the accuracy is improved compared with the statistical similarity method.Secondly,in the field of professional knowledge,the field of subject division has been the focus of attention of various industries.In this paper,the LDA topic model is introduced into the field of traffic text to achieve the topic partition of the traffic field and automatic generation technology of traffic terms.The traditional domain word classification has low real-time performance and low accuracy.Traffic terms can only be generated manually.In this paper,the all problems are solved successfully. |
