Neural Network Learning For Statistical Machine Translation

Research on statistical machine translation (SMT) has witnessed rapid growth in recent years, leading to substantial improvement in translation quality. However, the limited amount of bilingual training data, together with the lack of effective features, have impeded further progress, affecting various key components such as word align-ment, reordering and translation modeling. Meanwhile, deep learning, as an emerging machine learning method, can automatically extract abstract feature representations, modeling complex mappings between input and output signals. This new powerful technique opens up new avenues for SMT research. In this thesis, we will explore how to leverage deep neural network to learn better representation for translation modeling.Specifically, this work mainly consists of the following three aspects:●We propose a new deep neural network for word alignment modeling. We com-bine a multilayer neural network with a undirected probabilistic graphical model, accurately modeling word alignment by automatically exploiting lexical similar-ity and context similarity. We explore both semi-supervised and unsupervised training method for word alignment model. Large scale experiment on Chinese-English alignment task has confirmed the effectiveness of our method.●We propose a neural network based reordering model for SMT. Using a neural net-work based dimension reduction technique, we learns low-dimensional embed-dings for arbitrary reorder features; through a multi-layer network, these feature embeddings are integrated with word embedding features into a linear-ordering reorder models. Experiments on Chinese-English and Japanese-English show the proposed method significantly improve over strong baseline systems.●We propose a new network structure, recursive recurrent neural network, for translation modeling. Recursive recurrent neural network combines the strength of recursive and recurrent neural network, which not only can leverage arbitrary global features, but also can dynamically generate abstract representations for translation derivation tree. We apply this model to translation decoding for SMT, and propose a three-step training method for our model. Furthermore, we also investigate methods for translation pair embedding, proposing a translation con-fidence based method. Experiment on Chinese-English translation task exhibits strong improvement by using our method.In short, this work has investigated neural network learning for three main tasks in statistical machine translation. For each task, we have designed special neural network structures and learned task-specific feature representations. In future, we hope to merge all the representations into an unified abstract feature representation by exploiting neural network and SMT resources, and apply the learned features for other natural language processing tasks.