Variational Multimodal Machine Translation Research

Multimodal machine translation(MMT)aims at integrating image,voice and other multimodal data into machine translation algorithm.MMT help to eliminate ambiguity in pure text data,supplement machine translation data set in low resource scenes,and enhance the performance and robustness of machine translation system.However,the ambiguity of language leads to the semantic uncertainty in data,and there exists redundant information in multimodal data.These problems affect the performance of multimodal machine translation systems.Capturing the latent semantic relationship of sentences is helpful to machine translation.Variational neural machine translation(VNMT)provides an effective method for modeling the uncertainty of latent semantics in language by introducing latent variables.In this paper,a new MMT algorithm,called variational multimodal machine translation,is proposed under the variational framework.The algorithm models the semantic uncertainty in visual and text information.It only relies on the source data to construct the variational distribution,which can eliminate the difference between the training and the prediction in the existing VNMT techniques.To fuse multimodal information,the proposed MMT algorithm is designed on multitask learning,which can unify the multimodal semantic space with semantic prior distribution and constrains the distance from semantic posterior to semantic prio.In addition,the information bottleneck theory is used in the variational encoder-decoder framework to help the encoder filter redundant information and make the decoder concentrate on useful information.Finally,a self-supervised MMT algorithm is implemented by introducing contrast learning mechanism into the variational models.It can learn high-quality latent semantic features and adapt to the low resource machine translation.The MMT experiments show that the proposed algorithm is competitive in German to English translation tasks.