Research On Lattice To Sequence Neural Machine Translation

Neural Machine Translation(NMT)is an end-to-end deep learning approach for the problem of machine translation.Recent years have witnessed the rapid development and great success of NMT.Capable of modeling long distance dependencies,NMT has replaced the conventional statistical machine translation and become the current state-of-the-art machine translation method for both academic and industry.The dominant NMT models usually resort to word-level sequence modeling to embed input sentences into semantic space.However,it may not be optimal for the encoder modeling of NMT,especially for languages where tokenizations are usually ambiguous.On one hand,there may be tokenization errors which may negatively affect the encoder modeling of NMT.On the other hand,the optimal tokenization granularity is unclear for NMT.In order to alleviate these problems,we propose lattice to sequence neural machine translation models in this work.The main contributions of our work can be summarized as follows:(1)Deep attentional neural network for sequence labeling.We present a simple and effective architecture for the sequence labeling problem which aims at reducing tokenization errors.The model is based on self-attention that can directly cap-ture the relationships between two tokens regardless of their distance.Further experiments on semantic role labeling task also confirmed the effectiveness of the proposed architecture.(2)Lattice-based Recurrent Neural Network(RNN)encoders.We propose lattice-based recurrent neural network encoders which generalize the standard RNN en-coders to lattice topology.Specifically,they take as input a word lattice which compactly encodes many tokenization alternatives,and learn to generate the hid-den state for the current step from multiple inputs and hidden states in previous steps.Compared with the standard RNN encoder,the proposed encoders not only alleviate the negative impact of tokenization errors but are more expressive and flexible as well for encoding the meaning of input sentences.(3)Deep lattice to sequence neural machine translation models.To further enhance the expressive power of NMT,the lattice to sequence models are further extended with deep RNNs.We explore deep stack RNNs as well as deep transition RNNs in both lattice-based encoders and sequential decoder.By using deep neural net-works,the models can benefit from more flexible representations than the shallow ones.The above methods are validated on several tasks,including joint word segmenta-tion and POS tagging as well as Chinese-English translation.Experiments show that our methods can both reduce the tokenization errors and its negative effects on translation.It also demonstrates the superiorities of lattice to sequence models over the sequence to sequence baselines.