Research On Interpretability Of Neural Machine Translation:Model’s Representation,Training And Behavior

Recent years have witnessed great successes of Neural Machine Translation(NMT)both as a new machine translation paradigm that has developed tremendously and as an application in many language-oriented technologies such as online multilingual translation systems,simultaneous interpretation systems,and real-time chatting assitants.The stan-dard NMT framework consists of three essential componentsa).neural network based model architecture(with learned representations),b).effective sequence-to-sequence training(often with data augmentation),and c).beam search inference strategy for ap-proximately finding the optimal prediction.Despite undeniable progresses of NMT,the whole framework is known to be extremely difficult for both researchers and end users to interpret,because of the complex model architecture it adopts,with a growing number of trainable model parameters for scaling-up with big data.The hardness of interpretability of the NMT framework will lead to two barriers for future research and deployment of these large-scale NMT systems1).researchers could hardly interpret the inner working mechanisms of NMT models,thus they could not locate and find the cause of its weak-nesses and issues for improvements of the framework;2).end users could hardly interpret the inference process of NMT models,being unaware of reasons for the anomalous behav-ior,thus they might not trust the systems.This dissertation is motivated from the above discussion of the critical role of interpretability for NMT framework.This dissertation conduct several interpretability studies in terms of the three essential components of the framework,that is,this dissertation focuses on a).representationinterpretability[RI] to interpret the learned representation within complex model architecture,b).training interpretability[TI]to interpret the learning of data augmentation and unsupervised NMT,and c)behaviorinterpretability[BI]to interpret the relationship of model inference and model generalization.1.Interpreting and improving hidden representations of NMT[RI].Multilayer encoder-decoder architectures are currently the gold standard for large-scale NMT models.Existing works have explored some methods for understanding the hidden representations especially focusing on the encoder-side,however,the decoder’s representation is seldom explored and they have not sought to improve the translation quality rationally according to their interpretations of the learned representations.Towards interpreting for performance improvement,this dissertation first artificially constructs a sequence of so-called nested relative tasks and measure the feature generalization ability of the learned decoder’s representation over these tasks.Based on the interpretation,this dissertation proposes to regularize layer-wise representations with all tree-induced relative tasks.To overcome the computational bottleneck resulting from the large number of regularization terms,this dissertation designs efficient approximation methods by selecting a few coarse-to-fine tasks for regularization.Extensive experiments on two widely-used datasets demonstrate the proposed methods only lead to small extra overheads in training but no additional overheads in testing,and achieve consistent improvements(up to+1.3 BLEU)compared to the baseline translation model.2.Interpreting data augmentation in NMTtwo perspectives towards model’s generalization ability[TI].Many practical Data Augmentation(DA)methods have been proposed for NMT training in recent years.Existing works measure the superiority of DA methods in terms of their performance on a specific test set,but this dissertation finds that some DA methods do not exhibit consistent improvements across different translation tasks.Based on the above observation,this dissertation attempts to answer a fundamental questionwhat benefits,which are consistent across different translation tasks and specific DA methods,does DA in general obtain for the model?Inspired by recent theoretic advances in deep learning,the dissertation interprets DA methods from two perspectives towards the generalization ability of a modelinput sensitivity and prediction margin.These two metrics have less dependence on specific test sets thereby may lead to findings with relatively low variance compared to BLEU which also depends on specific target references.Extensive experiments on four translation tasks with five DA methods show that the proposed two metrics have better consistency compared to BLEU,so they might be used as novel intrinsic evaluation metrics for DA methods.3.Demystifying learning of unsupervised NMT[TI].Unsupervisedly training the NMT model(UNMT)has received great attention in recent years.Although tremendous empirical improvements have been achieved,there still lacks theory-oriented investigations and therefore some fundamental questions like why a certain training protocol can work or not under what circumstances have not yet been soundly explored and understood.To this end,this dissertation attempts to provide several theoretical insights for the above questions.Specifically,following the methodology of comparative study,this dissertation adopts ideas from two theory-grounded perspectives,i)marginal likelihood maximization and ii)mutual information from information theory,to interpret the different learning effects from the standard training protocol and its variants.Our comparative studies reveal a few critical conditions for successful trainings of UNMT and some valuable findings.4.Detecting source contextual barriers for interpreting model behavior of NMT[BI].In machine translation evaluation,the traditional wisdom measures model’s generalization ability in an average sense,for example by using corpus BLEU.However,the statistics of corpus BLEU cannot provide comprehensive interpretations and fine-grained analyses on model’s generalization ability.As a remedy,this dissertation attempts to interpret NMT at fine-grained level,by defining and detecting the so-called source contextual barriers within an unseen input sentence that cause the degradation of model’s translation performance.The dissertation proposes a principled definition of source contextual barriers as well as its modified version which is tractable in computation and operates at word-level.The modified definition leverages word editing to generate counterfactual sentence-level metric values for computing the risk of each source word of being a barrier word.Based on the modified definition,three simple methods are proposed for estimating the risk,so as to detect the words with highest risks as the so-called source contextual barrier words.The dissertation conducts experiments on the Zh_→En and En_→Zh NIST benchmark,and carefully analyzes the detected barrier words with respect to part-of-speeches,other source word categorizations,contextuality etc.The usage of barrier words in re-ranking and human model debugging are discussed quantitatively and qualitatively respectively.