Research On Modal Adaptation For Image Description Translation

Image Description Translation(IDT)takes a source language description and translates it into the target language,where this process can be supported by information from the image.Image descriptions are mainly short texts,which can not provide enough contex-t information for translation system.Therefore,the focus of this task is how to integrate image information into translation system for solving cross-modal information fusion prob-lem.This paper concentrates on the optimization method of IDT with strong adaptability on Statistical Machine Translation(SMT)and Neural Machine Translation(NMT).In SMT,aiming at the problem of ambiguous words in corpus,this paper optimizes translation model through topic information of the image,so as to enhance the domain adaptive ability of the system.In NMT,this paper improves the performance of translation system by optimizing the supportive role of image features for learning language features.This paper focuses on IDT,including the following contents:(1)Translation model optimization for fused image topic informationThe general domain translation model cannot accurately translate ambiguous words.To solve the problem,we propose a method to optimize translation model based on image topic information.This method aims at mining documents similar to the image from large-scale image-document corpus to analyze the topic information of image.Then the topic informa-tion of images is integrated into the translation model of SMT to enhance domain adaptivity and improve performance of system.Experimental results show that our method increases the translation performance by nearly 0.74 BLEU points compared with baseline.(2)Relevant sentences extraction for topic optimization in Image Description Transla-tionAiming at the problem of analyzing image topic,we propose a method of extracting relevant contexts and a method to optimize image topic distribution based on contexts.The study aims to obtain a set of sentences or paragraphs closely related to the image from a collection of documents obtained during the image matching process.We use obtained paragraphs or sentences to analyze image topic distribution.Then the topic information of images is integrated into the translation model of SMT to enhance domain adaptive ability.Experimental results show that the method can improve the performance by 1 BLEU points compared with baseline.(3)Multi-channel and Multi-modal Image Description TranslationIn NMT,we propose a multi-channel and multi-modal image description translation model for solving the problem that image features are not suitable for serialization tasks.The method optimizes image features from image classification model by pre-trained Im-age Description Generation(IDG)model.And then,we use image features to initialize the hidden state of encoder?decoder and add it into encode-decode attention mechanism,so as to improve the performance of IDT.Experimental results show that the proposed method can effectively utilize image features and improve the performance by 6 BLEU points on Multi30k-16?Multi30k-17 and Ambiguous COCO in the best case.This paper focuses on solving the modal adaptation in IDT.In terms of SMT,this paper attempts to solve the problem of multi-modal information fusion by domain adaptation;In the aspect of NMT,this paper proposes a multi-channel multi-modal translation method for the problem that image classification features are not applicable to serialization tasks.This method focuses on the multi-modal information fusion problem in the decoding.By analyzing experimental performance and translation examples,this paper demonstrates the effectiveness of the above methods in solving modal adaptation problems.