Transition-based Dependency Parsing By BiLSTM And Attention Mechanism

Dependency parsing is an important task in natural language processing.There are two major methods of dependency parsing: one is graph-based dependency parsing,the other is transition-based dependency parsing.Existing transition-based dependency parsing models mostly employ BiLSTM which is capable of capturing bidirectional long-distance dependency in sentences,but following problems exist: 1)The models are complex and computaionally intensive which lead to long training time;2)The accuracy of dependency parsing is low when the distance between the head word and the dependent word is large.To deal with above problems,the main work of this thesis is as follows:(1)Aiming at the problem of long training time,a BiLSTM-TB(Transition-Based dependency parsing model by BiLSTM)model is proposed based on the classic BiLSTM model.The feature function is improved and simplified which only contains the first two words in the stack and the first word in the buffer.We conduct experiments on PTB and CTB data sets.The experimental results show that compared with the classic BiLSTM model,the BiLSTM-TB model achieves similar dependency parsing accuracy,while the training time is reduced by 28.3%.(2)Aiming at the problem of low dependency parsing accuracy when the distance between the head word and the dependent word is large,we add attention mechanism to BiLSTMTB model and propose BiLSTM-Attention-TB(Transition-Based dependency parsing model by BiLSTM and Attention)model in this thesis.Attention mechanism can obtain more information in the sentence,even when the distance between the head word and the dependent word is large.We conduct experiments on PTB and CTB data sets.The experimental results show that BiLSTM-Attention-TB model achieves 93.2% for unlabeled attachment score(UAS)and 90.8%for labeled attachment score(LAS)on PTB data sets and achieves 87.0% for UAS and 84.5%for LAS on CTB data sets.Compared with the BiLSTM-TB model,the BiLSTM-Attention-TB model improves on both UAS and LAS,and performs better when the distance between the head word and the dependent word is large.Specificly,when the distance is between 3 and 6,the accuracy improves 0.2%,when the distance is great or equal to 7,the accuracy improves 0.3%.Besides,we conduct experiments on PTB and CTB data sets to analyse the effects of different dependency relations and parts of speech on the performance of BiLSTM-TB model and BiLSTM-Attention-TB model.The results show that the model performs well on the dependency relations and parts of speech with certain parterns,and poorly on the dependency relations and parts of speech with flexible forms.The use of external word embeddings can improve the accuracy of the model.When the dimension is 100,the model achieves highest accuracy.The ablation study of BiLSTM-Attention-TB model show that the BiLSTM is the core part of the model,and Attention mechanism needs to work on the basis of BiLSTM.