Research And Application Of Self-attention Mechanism In Semantic And Sentiment Analysis

With the development of information science and technology,Natural Language Processing technology has gone through the stage from statistical machine learning to neural network learning,and then the application of attention mechanism has made great progress in information extraction,machine translation and other issues,and it has also been widely used in the monitoring of public opinion in enterprises.In the enterprise's decision-making and development,public relations executives need to analyze the company's word of mouth information over the past period of time and predict the company's development and tendencies in the future based on public opinion data on the Internet.In combining attention with Natural Language Processing technology,it is faced with many problems that need to be solved.If the public opinion text data for Internet companies and the public opinion data for financial enterprises expressed the text information subject,different ways of text processing into structured data may lead to the lack and ambiguity of information in completeness,which is an urgent problem to be solved in language coding.Attention mechanism was first proposed in the field of Computer Vision,people began to study similar attention mechanisms to perform simultaneous translation and alignment on machine translation tasks subsequently,which was the first attempt by attention mechanism in the field of Natural Language Processing.Later,attention mechanism is widely used in various Natural Language Processing tasks based on recurrent neural networks and convolutional neural networks,and self-attention is a special case in attention mechanism,which plays an important role in extracting text features.At present,the attention model is used in combination with the Encoder-Decoder,which is a research pattern in the field of deep learning,and has a wide range of applications.This thesis proposes a semantic understanding method of self-attention based on Encoder-Decoder,which can monitor public opinion of enterprises by analyzing the public opinion text data in depth.First of all,for the processing of text data,the thesis gives a mixed language representation method named Deep Excavation Language Model.In the process of word vector learning,by studying the traditional Continuous Bag-ofWord model,we found that some shortcomings in it,in order to improve the presentation of language,we use long-short term memory network to learn temporal relationship between words based on the bag of words,the also introduction of knowledge of entity recognition in text at the same time helps improve the presentation of words.In specific fields,certain words have special meanings,and in order to fully understand and learn the semantics of words in specific contexts,the corresponding entity knowledge can enhance the learning process of words and convert them into a vector representation in a specific space.Secondly,in view of the complexity of feature extraction in text,the thesis introduces an Encoder-Decoder model based on self-attention mechanism,which solves the problem of end-to-end text sequence processing.This method combines the selfattention mechanism with the traditional Encoder-Decoder,fully learns the representation characteristics of the whole statement,completes the expression and learning of the characteristics at the syntactic level,and the characteristics of its extraction are closely related to the specific tasks downstream.Finally,in order to monitor public opinion,a public opinion scoring model based on neural networks is given in the thesis.The model uses the Deep Excavation Language Model and self-attention-based Encoder-Decoder model proposed in this thesis to evaluate the sentiment of public opinion.It is proved that the language representation method and text feature extractor proposed by the thesis have significant effects in the analysis of public opinion data by experimental comparison and analysis.