Improving Neural Machine Translation With Syntactic Knowledge

Neural machine translation(NMT)with the attention-based encoder-decoder framework has achieved significant improvements over the statistical machine translation in terms of fluency and fidelity.In a conventional NMT model,an encoder reads in the source sentences of various lengths word by word and transforms the input words into intermediate hidden states.After weighted by the attention model,the decoder uses the hidden vectors to generate the target sentences.However,the construction of a sentence in any language is not a simple list of words.In fact,each sentence is constrained by the inherent syntactic structures in a certain language.For an English sentence,it is usually composed by the subject,the predicate and the object and each English word can play one or multiple roles in a sentence.With the restriction of the syntax,the words could build a correct sentence.But the NMT usually neglects the syntax which leads to translation errors violating the source or target syntax.Recently,the syntax-based NMT has been extensively studied.In this thesis,we propose to explore the following five topics in syntax-based NMT.First,we propose a simple but effective source dependency based NMT model.Source syntax can help the NMT encoder better understand the structure of the source sentence,such as the subject-predicate and the predicate-object.With this knowledge,the decoder may generate more accurate translations.Many methods have been proposed to incorporate source syntax into NMT.But they usually leverage complex neural networks to model the source syntactic tree,which makes the more hard to train.Our proposed method linearize the source dependency tree by different traversal.We use the post-order and the pre-order traversal to construct two extra sequences and let the NMT encoder to encode them.The two sequences keep the structure knowledge of dependency trees.Thus we build a source syntax-aware encoder for NMT.Experimental results show that our method could significantly improve the translation quality.Second,the neural Transformer has significantly surpassed the NMT models based on the recurrent neural networks.However,there is no research work to verify whether the syntax is useful for the Transformer model.In this thesis,we propose a source dependency-aware encoder for the Transformer.The Transformer uses the multi-head selfattention to encode the source sentence,which implicitly models the source sentence from different views.Based on the self-attention network,we propose two dependency matrices according to the source dependency trees.The two matrices model the dependency tree from head-to-child and child-to-head.During the model training,we use the two matrices to supervise the self-attention,thus the Transformer could learn the source dependency tree explicitly.Experimental results show that the proposed method could effectively use the source dependency trees and significantly improve the translation quality.Third,we propose a sequence-to-dependency neural machine translation model to leverage the target-side dependency trees.On the target-side,syntax could help NMT generate more grammatical translations.Existing methods fail to use the target syntax to help translations.Our sequence-to-dependency NMT model can jointly conduct translation and dependency parsing.During the translation,we extract syntactic context from the partial dependency trees,which is used to facilitate the future generation of target words.Our experiments show that the sequence-to-dependency model can significantly improve the translation quality with help of target trees.In addition,our model can generate reasonable target dependency trees.Forth,we propose a dependency-to-dependency neural machine translation model to use both sides' dependency trees.Syntax is much more complex than words.Using a single side syntax is non-trivial,so it is more challenging to simultaneously use both syntax structures.Our novel dependency-to-dependency NMT model achieves this based on the former work.We prove that the effect of source and target syntax can be accumulated.Fifth,in this thesis,we do analysis our syntax-based NMTs from three perspectives:(1)the effect of the training scale on the syntax-based NMTs;(2)the effect of the parsing qualities on the syntax-based NMTs;(3)The effect of the syntax-based NMT on both similar and non-similar language pairs.