The Application In Text Retrieval Of Semantic Similarity Calculation Based On The Neural Network

In recent years, the natural language processing becomes more and more popular, the semantic similarity calculation research has become an important branch of the field. Its application in information retrieval is very extensive, and it plays a key role in the syntactic analysis. There are many outstanding scholars exploring in the semantic similarity calculation. It has been more than 10 years from put forward, the definition of it is becoming more and more clear. From the end of last century, the domestic and foreign scholars on the research has two methods to research it. The first method is calculating according to the mass of knowledge, the main idea is based on the concept of the words hierarchy of the semantic dictionary method,handover with a word through the up and down link to do similarity calculation. The second method is using the related context-sensitive semantic information selected through the feature of statistical crawl corpus, and calculate probability distribution between keywords and the context words which have stronger relation with them. Now joined the neural network model to train the words, in order to get better words' semantic and grammatical information, and get better word vectors, the classic models are CBOW and Skip-gram model. This article is combining multiple models and optimizing the algorithm to get a better method of words similarity calculation, good calculation results have been achieved.The text retrieval which is very important in retrieval plays an important role in many aspects of education and life. This is a new era and people can get the vast amounts of information. However, in the current situation. On the one hand, we can get more information in the case of the large data. And on the other hand, because of the background of the large data. Information on the network becomes complex. This not only makes it hard for information management, but also adds the difficulty of managing and finding high value information and resource. So, tools which allow users to rapidly obtain information they need have become the urgent needs of us.The article researches the application in text retrieval of semantic similarity calculation based on the neural network. The application is embedded in improve the text retrieval accuracy and recall rate of ascension, and comprehensive evaluation index.Combination with the previous semantic similarity algorithm, I put forward a new improved algorithm: improved version of the semantic similarity calculation based on neural network. This paper mainly do the following work:(1)Introducing the development and the research of semantic similarity computation,elaborating the emerging of semantic similarity calculation based on the neural network and the improved algorithm. Briefly explaining the concept of retrieval system, the core technology, and the research introduced similarity calculation.(2)The data of Wikipedia crawler, Chinese words segmentation, writing programs for texts segmentation, using the predecessors' experience in semantic similarity calculation to improve the algorithm, design better training model to train the word vector, get more accurate data of semantic similarity. Retrieving segmentation texts without similarity calculation; Finally, the Similarity data comes out from texts similarity calculated put into the custom similarity formula of the Lucene,then retrieving the segmented texts.Comparing data, showing its algorithm design is helpful to improve retrieval ability.