Question Answering Model Based On Self-Attention Mechanism

Machine reading comprehension is one of the hottest topics in natural language processing.Especially with the rise of deep learning,machine reading comprehension algorithms have also been fully developed.Existing machine reading comprehension tasks can be classified into four categories: fill-in-the-blank reading comprehension tasks,selective reading comprehension tasks,extractive reading comprehension tasks,and generative reading comprehension tasks.Extractive tasks are the most frequently studied reading comprehension tasks in recent years.It needs to find answers in a large amount of text,and there will be some plausible answers in the text.Compared with other tasks,the problems faced by extractive tasks are also more difficult.This paper mainly focuses on the extraction reading comprehension task,selects the SQu AD dataset as the object for research,and extends the algorithm for the newly added unanswered questions in the SQu AD2.0 dataset.Machine reading comprehension algorithms can be roughly divided into two categories: methods that use pretrained models and methods that do not.In pre-training methods,fine-tuning of downstream tasks is mostly adopted.The method of not using pre-trained models,although not as good as the former,can provide researchers with more room for imagination in model innovation,and promote us to explore the meaning behind different structures and models.This paper attempts to focus on the structure of the reading comprehension model itself,instead of using pre-training models,and abandoning the recurrent neural network structure most of the mainstream models use,using the self-attention mechanism to build the coding layer and build a machine reading comprehension model.Specifically,based on the QANet model,this paper incorporates the relative position encoding into the encoder structure,introduces the start position feature when predicting the end position of the answer,and extends the model to include "no answer" types of reading comprehension tasks.An improved question answering model based on self-attention mechanism.The main innovations of the model include:1.In view of the insufficiency of fixed absolute position encoding,a relative position encoding scheme is proposed and introduced into the self-attention mechanism module in the encoding layer and modeling layer of the model.2.Introduce a new structure in the output layer to establish the connection between the starting position and the ending position,so as to eliminate the isolation of the output starting position and ending position of the original model.3.For the task of judging whether the question can be answered in the SQu AD2.0 dataset,a classification module is added to the output layer,and multi-task training is adopted.In order to verify the effectiveness of the proposed algorithm,experiments are carried out on the SQu AD dataset.The experimental results verify that the relative position encoding scheme proposed in this paper has obvious improvement compared with random initialization position encoding and absolute position encoding;on the SQu AD1.The EM and F1 scores are increased by 3.2% and 3.9% respectively;on the SQu AD2.0 dataset,which adds the judgment of whether the question can be answered,the method in this paper is compared with the best comparison algorithm Doc QA+ELMo,in the EM and F1 scores.were increased by 2.2% and 3.1%.This shows that our proposed method has good performance for extractive reading comprehension.