Sentence Representation Research Based On Attention Mechanism

Sentence representation is the basis of different tasks in natural language process-ing,which makes a sequence of words as digital information that can be processed by the computer.A good sentence representation needs to contain the grammatical and se-mantic features,but how to accurately represent the grammar and semantics by numbers has always been the biggest challenge in the field of natural language processing.In recent years,the rise of deep learning has made the sentence representation more popular.The sentence representation methods based on attention mechanism have the abilities of flexible modeling,good interpretability and efficient computing,which is favored by a lot of researchers.However,there are two problems in the existing sentence representation methods based on the attention mechanism:1.The attention mechanism method is a weighted sum method,and the relative positional relationship of the words is not considered,that is,the sentence structure information is missing;2.There are many basic calculation methods for attention mechanisms,and the differences are huge,which makes the design of model more cumbersome.In this paper,we mainly study the sentence representation method based on the attention mechanism.We summarize our work into the following two points:(1)We proposed Multiple Positional Self-Attention Network(MPSAN)for sentence representation.In MPSAN,different masks are applied to the attention mecha-nism to extract various sentence representations,and the Fusion Mechanism is designed to integrate multiple sentence representations into one sentence repre-sentation.This can introduce the relative position information of the words into the attention mechanism,correct the weighted sum method,and solve the problem that the attention mechanism lacks the sentence structure information.Specifi-cally,we designed Faraway Mask and Scale-Distance Mask,which could extract the local information of the sentence,and we used the Forward Mask and Back-ward Mask to extract the order information of the sentence.In addition,in the Fusion Mechanism,we designed a sharing strategy of parameters,which made the number of parameters from square level to linear level.We evaluated our MPSAN on the sentiment analysis task and text classification task.The experi-mental results show that our MPSAN not only has an advantage in time and space complexity,but also obtains a better test accuracy.(2)We proposed Parameter-Adjustable Attention Network for sentence representa-tion.We summarized the various variants of the attention mechanism from the perspective of parameter level,and found a better design method of attention mechanism.We designed two new compatible functions in our PAAN:constant compatible function and mixed compatible function.Among them,the mixed compatible function considers the vector combination of splicing and multiplica-tion.We evaluated our PAAN on the Stanford Sentiment Treebank(SST).The experimental results show that the mixed compatible function has a better ability to extract information than other compatible functions,and the multi-dimensional method can make the improvement in attention mechanism.