Controllable Image Synthesis And Manipulation Based On Deep Generative Model

Controllable image synthesis and manipulation can be applied in many computer vision related scenarios like image processing,super-resolution,etc.At present,most researches mainly focus on unconditional image generation tasks and are dedicated to improve the training stability and generation quality in terms of fidelity and diversity.Controllable image synthesis and manipulation still lack in-depth research compared with unconditional generation researches.Moreover,the black box property of deep generative models,low fusion efficiency for conditional inputs,need for great amount of labeled train samples,low interpretability of latent space,all these problems impose great challenge for controllable image synthesis and manipulation.This dissertation aimed at exploring effective controllable image synthesis and manipulation methods,through which deep generative models can be controlled and produce desired results concord with conditional inputs given by users.The main contributions and innovations of this dissertation are listed as below:This dissertation first investigated conditional inputs-driven controllable image generation models,and proposed effective controllable image synthesis methods for two type of conditional inputs(one was discrete categorical input and the other was contin-uous feature values input).For continuous conditional inputs controlled image genera-tion,this dissertation proposed an effective controllable eye image synthesis method by adopting an auxiliary regressor and proposed a simple but effective strategy for stabilize the train process.For categorical conditional inputs controlled image generation,this dissertation proposed to improve model architecture by introducing an auxiliary feature extractor in discriminator,which was demonstrated can improve the model performance on conditional generation.This dissertation further studied conditional inputs-driven controllable image ma-nipulation methods,and proposed an effective facial attributes editing model based on attribute differences of unpaired face images.For face image attributes editing,this dis-sertation introduced an attribute difference extraction model base on Siamese network,through which attribute difference for any unpaired face images can be effectively ex-tracted and manipulation process can be controlled.The proposed method can handle single-attribute and multiple-attributes editing tasks effectively,moreover,when the la-beled train data was in shortage,the proposed method can boost data utilize efficiency,thus the performance of attribute editing models can be improved.This dissertation also explored the interpretability of the latent space of deep gen-erative models and fulfilled interpretable analysis of the latent space of deep generative models by quantifying the importance of different latent dimensions to specific semantic concept generation.Moreover,this dissertation also proposed two methods for locating key latent dimensions with high correlation of specific semantic concept generation,one by sequential intervention and the other by optimization.Experimental results proved that the proposed method can help finding high-correlated latent dimensions for specific semantic concept generation and fulfill controllable concept manipulation through in-tervention on the located key latent dimensions.This work also provide a new research route for fulfilling controllable image synthesis.