Single And Multiple Modal Image Transformation Based On Generative Adversarial Learning

In recent years,various researches on image processing have received extensive attention and development.In the field of computer graphics,image transformation is an important technology,and many image processing problems can be attributed to this task,such as image style transfer,image denoising,image super-resolution and so on.The purpose of image transformation is to study the relationship between domains and to solve the problem of how to transform an image of one modality into another modality.Therefore,image transformation needs to focus on the similarity and diversity of images.The content of similarity requires both the structure similarity of the generated image and the input image,as well as the detailed similarity between the generated image and the sample of the target image domain.The content of diversity requires that the method has good scalability when processing multi-modal output or input images.According to whether the paired data exists,the existing image transformation methods can be divided into supervised method and unsupervised method.According to whether contain multi-modal images,methods can be divided into single-modal and multi-modal method.Paired training data provide ground truth to guide image generating,which reduces the difficulty of similarity requirement in research content.However,paired training data is difficult to obtain,so unsupervised method is more universal than supervised method.Compared with single-modal method,the multi-modal method is more complete,but the multi-modal method is more complicated in solving multiple modalities of the output images.Finally,the multi-modal method is often more complex when applied to multiple image transformation tasks,so how to reduce the parameters of network is a challenge.Aiming at the situation that there is no paired training data in image transformation method,this paper proposes a single modal image transformation method based on generative adversarial learning.This method utilize the generative adversarial nets to perform unsupervised image transformation and improve the generated image quality,meanwhile learn the bidirectional mapping between hidden space and data space under the similarity of generating image with input image and target domain image,the mutual information mechanism and perceptual loss are introduced to guide image generating.In the tasks of edges to photos,multiple face attribute transformations and face inpainting,the method proposed in this paper preserves the key representation of the input image while transforms the partial characterization into other modality.In the multi-modal image transformation method,on the one hand,one image may correspond to multiple outputs,aiming at the problem of how to reducing the uncertainty of output image in multi-modal method,this paper proposes the multiple modal image transformation method based on generative adversarial nets.The method uses an additional encoder to generate the style vector,then guides the image generating through different style vectors.The method optimizes the style vector and image generating by the mutual information mechanism,perceptual loss and cycle consistency loss,which providing the scalability of model.On the other hand,for reducing the training parameters of multi-modal method,this paper considers the solution in the case of multi-modal inputs.The method adds a condition vector for each modality in the model and combines the inference network of the variational autoencoder and the adversarial network of the generative adversarial nets.The proposed method reduces the training parameters,the structure is more concise,and good performance has been obtained in the experimental results.