Structural Information Integration For Neural Machine Translation

Recent years have witnessed the great success of end-to-end Neural Machine Translation(NMT),which has outperformed traditional Statistical Machine Translation(SMT).End-to-end NMT adopts the Encoder-Decoder architecture to model the translation procedure.The Recurrent Neural Network(RNN)encoder encodes the source word sequence into a continuous representation,based on which the decoder generates a target word sequence.However,the disadvantage of modelling source and target sentences in the sequence form lies in the lack of structural information,leading to the inadequate translation problem.This thesis explores introducing structural information into NMT.We propose to integrate word alignment structure,phrase structure and syntactic structure into the NMT decoder.The major research directions can be divided into three parts:(1)Word alignment structure integration for NMTOn the word level,NMT produces fluent but inadequate translations due to the unawareness of word alignment structure.To address this problem,on the word level we propose a framework integrating word alignment structure.Under this framework,issues caused by the lack of word alignment structure can be further alleviated by leveraging external word alignment information at each decoding step.Specifically,we regard word alignment structure provided by SMT as external alignment information and introduce it into the decoding step of NMT.During the decoding stage,SMT provides appropriate word alignment information based on the decoding information of NMT,and further perform word recommendations in order to guide the NMT decoder to estimate probabilities of the target vocabulary more accurately.Experimental results show that NMT models that integrate word-level knowledge generated from SMT can effective learn word-level knowledge from SMT to improve the translation performance.(2)Phrase structure integration for NMTOn the phrase level,we propose a framework that integrates phrase structure,under which NMT decoder can receive external phrase knowledge at each decoding step to alleviate the missing of phrase structure.Specifically,we propose to leverage phrase knowledge from SMT as external knowledge and introduce it into the decoding step of NMT,which can help the phrase generation procedure for NMT.However,NMT decoder generates the target sequence based on word units,leading to inconsistencies between phrase generation and word generation in terms of granularity.We propose a method that integrates a phrase memory device into the traditional Encoder-Decoder architecture and designs a generation model which is consistent between word and phrase levels.During each decoding time step,SMT provides phrase recommendations based on decoding information generated by NMT and writes into the phrase memory device,based on which NMT determines whether to generate a new phrase.When generating a phrase,NMT reads phrases stored in the phrase memory device and selects the appropriate phrase as an output.Experimental results show that the phrase memory device configuration can help NMT to appropriately generate phrases and effectively improve the translation quality of NMT.(3)Syntactic structure integration for NMTWe further propose a framework to integrate syntactic structure following the integration of phrase structural information.Specifically,we propose an NMT translation model that can capture syntactic structure based on syntactic structure skeletons.Based on the Encoder-Decoder architecture,the proposed model consists of two sub-decoders: the skeleton decoder and the attribute decoder to capture the syntactic structure information for target sentences.We propose a two-pass approach leveraging these two sub-decoders to construct target sentences merging syntactic structure.The skeleton decoder generates theskeleton of the target sentence first,based on which we leverage the attribute decoder to generate the target sentence.Experimental results show that the syntax-based skeleton NMT model can automatically perform the phrase generation and improve the translation quality.With the help of integrations of word alignment structure,phrase structure and syntactic structure,we gradually complete the modelling of structural information on different levels.We hope to see that preliminary results in this thesis can guide future research of structural information and motivate the development of machine translation research.