Research On Attention Neural Network And Its Application In Natural Language Understanding

Natural language understanding is a subject that studies how computers understand and process natural language data,including many subtasks such as machine translation,textual question answering,sentiment classification.In the field of natural language processing,many traditional machine learning algorithms,such as support vector machine,Gaussian mixture models,and random forests,have been well applied.In recent years,deep learning has received increasing attention.Among them,deep neural networks have been greatly developed and successfully applied in many fields,such as natural language understanding,image processing.A neural network can be regarded as a network composed of highly connected neurons(cells).Considering the characteristics of every task in natural language understanding,researchers proposed numerous neural networks with different structures.Recently,much work has shown that introducing attention mechanisms into neural networks can greatly improve their performance.In this study,neural networks with attention mechanisms are referred to as attention neural networks for short.Compared to neural networks with average attention,attention neural networks allow attention to be distributed to elements of the signal as needed.Attention mechanisms in neural networks also accord with the process of thinking when humans solve problems.This study mainly explores the natural language understanding algorithm based on attention neural networks.It focuses on textual question answering,text classification,and multi-domain sentiment classification in natural language understanding.The main research work and innovations are summarized as follows:(1)Aiming at the problem that the existing dynamic memory network DMN only considers the single type of interactive features of the input facts and questions when completing the textual question answering task,and cannot fully simulate the multiple logical associations between the input texts,a textual question answering method based on a dynamic memory network with dual features En DMN is proposed.In the process of designing the dynamic memory network based on dual features En DMN,global feature and hierarchical feature extractors are introduced to extract the global features and hierarchical features of the input text respectively.The global features reflect the overall meaning of the input text,and the hierarchical features reflect the salient features of the input text that need to be focused on at each layer.At the same time,considering these two types of features can simulate various logics between the input facts and the question from multiple perspectives.In the feature extraction process,the differentiated network layers are introduced to more effectively extract the global features and hierarchical features of the question;in the feature generation stage,Atten GRU that can further integrate temporal information and attention weights are used to obtain more expressive feature vectors,and a textual question answering system is finally constructed based on the dynamic memory network with dual feature En DMN.Experimental results show that compared with other dynamic memory networks based on single features,the dynamic memory network based on the dual feature En DMN and its text question answering system can obtain the best average test accuracy in the b Ab I data set,which contains multiple textual question answering tasks.(2)Aiming at the problem of insufficient removal of redundancy and conflict information when the current attention neural network completes text classification tasks,a text classification method based on an attention neural network with multi-layer supervision AMMS is proposed.In the process of designing attention neural network with multi-layer supervision AMMS,the attention weight matrix on each layer is generated directly from the matching score between the previous context vector and the hidden state of the input text at each time step,and the feature vector representation of the input text on each layer is generated accordingly.The objective function considers the loss function at all layers where the context vectors of the input text are mapped to the prediction labels(IE,multi-layer supervision),which can ensure that the typical features required for classification are gradually extracted from the relevant information.A text classification system is constructed based on the attention neural network with multi-layer supervision AMMS.Experimental results show that compared with other hierarchical/multi-layer attention neural networks,AMMS can maintain similar computational complexity as that of single-layer attention neural network,while it can obtain similar or better performance for multiple text classification data sets.(3)Aiming at the problem that SAN cannot accurately extract domain-related sentiment features and DAM cannot effectively process long texts containing more noise signals and gain domain-aware hidden states when completing sentiment classification tasks,a multi-domain sentiment classification method based on the collaborative attention neural network CAN is proposed.CAN combines the respective advantages of SAN and DAM,through the joint use of a multi-domain sentiment classifier based on the domain attention mechanism and a general sentiment classifier based on the self-attention mechanism,to complete the main sentiment classification and auxiliary sentiment classification tasks respectively.The hidden state generated at each time setp in the self-attention module used to complete the auxiliary sentiment classification task is regarded as the input signal of the domain sub-module and the sentiment sub-module in the domain attention module to promote the sentiment sub-module to effectively process the input signals with less irrelevant information and generate domain-aware hidden states in a new context for multi-domain sentiment classification.In addition,by working with the multi-domain sentiment classifier,the general sentiment classifier can also obtain useful domain information of the input text.In this study,the collaborative attention neural network CAN is used to constructed a multi-domain sentiment classification system.Experimental results show that,compared with other multi-domain sentiment classification models and methods,the multi-domain sentiment classification method based on the collaborative attention neural network CAN can obtain the best overall performance on Amazon English and JD Chinese multi-domain sentiment analysis data sets.