| Text-to-image generation is a kind of cross-modal conversion technology and is a hot research topic in Computer Vision and Multimedia Community.The core task of text-to-image generation is transforming one-dimensional text information into twodimensional realistic images that conform to the semantics of the input text by correctly analyzing and encoding the input text.Text-to-image generation can meet people’s demand for diverse images.When people want to obtain images that conform to their requirements and imagine,they can get the images that do not exist on the Internet or databases based on language or text through text-to-image generation methods,which greatly reduces the cost of obtaining images.Therefore,text-to-image generation has a wide application prospect in computer-aided design,digital entertainment,social media,and other fields.Text-to-image generation methods can generate high-quality single-object images,such as images with a flower or a bird.But for the situation of text-to-scene image generation,there are still many unsolved problems.First of all,it is necessary to analyze the input text structurally,use the existing image materials flexibly,and optimize and adjust the scene image elements according to the constraints in the input text to generate realistic scene images with high resolution.Second,when modeling the corresponding relationships between input text and scene images,it is necessary to correspond each object in the scene image with all words that describe its name and properties.It is also essential to judge the correctness of relative positional relationships between different objects in the scene image.In addition,the discriminators need reasonable training and discriminating mechanisms to provide correct guidance for the training of the generators in Generative Adversarial Networks.Otherwise,problems of training instability and reduced diversity of the generated images may arise.To address the above problems,this dissertation proposes several effective solutions.First,a rule-based heuristic text-to-scene image generation model is proposed.This model includes text processing and analysis,foreground objects and background image retrieval,object locations and sizes optimization,and post-processing of the scene images.Second,a phrase-boost Generative Adversarial Network is proposed,which encodes the words that describe the same object in the input text into a phrase embedding to learn more accurate phrase-object correspondence and model the relative positional relationships between different objects,which helps to generate high-quality scene images that are more consistent with the input text.Finally,a triple-discriminator Generative Adversarial Network is proposed,consisting of three discriminative networks with different functions.These discriminators have proper training and discriminating mechanisms.They can provide correct guidance for the training of the generators,which improves the stability of the network training and the diversity of the generated images.Specifically,the main research contents and contributions of this dissertation are summarized as follows:(1)To resolve the problem of generating realistic high-resolution scene images,this dissertation proposes a comprehensive model that includes text processing and analysis,foreground objects and background image retrieval,object state optimization,and image post-processing.First,the model analyzes the input text and obtains the names of foreground objects and background scenes,and structurally represents the semantic relations between them.Second,the model retrieves appropriate foreground objects and background scene images from the foreground object dataset and background image dataset.Third,to ensure the rationality of the positions and sizes of all foreground objects in the scene image,this model designs a cost function according to the constraints and rules and uses the Markov chain Monte Carlo method as an optimizer to obtain a reasonable scene layout.Finally,this model uses Poisson-based and relighting-based blending methods in the post-processing step to fuse the foreground objects and background image into a harmonious and natural scene image.(2)To resolve the problems of modeling the correspondence between each object in the image and all words that describe it and modeling the relative positional relationships between different objects in scene images,this dissertation proposes a phrase-boost Generative Adversarial Network(Phrase GAN),which encodes all words that describe the same object in the input text as phrase embedding to learn more accurate phrase-object correspondence and model the relative positional relationships between objects based on the phrase embeddings.First,Phrase GAN uses a Transformer-based text encoder to encode the input text into the embeddings of words and sentences.Then the novel phrase embedding is computed based on the word embedding and text correlation analysis.Second,Phrase GAN builds a text-image similarity model based on Graph Convolutional Networks,which can evaluate the fine-grained phrase-object similarity between the input text and the generated scene image and model the relative positional relationships between multiple objects.Finally,a phrase-object discriminator is proposed to judge the quality of the generated objects and scene images.(3)To resolve the problem that the discriminator has unreasonable training and discrimination mechanisms,resulting in unstable network training and reduced diversity of the generated scene images(mode collapse),this dissertation proposes a Triple-discriminator Generative Adversarial Network(TDGAN).First,a diversitysensitive conditional discriminator is proposed,which can recognize mode collapse and increase the diversity of the generated images by judging the combination of the generated image and mismatched text as false.Second,an unconditional discriminator based on a contrastive searching gradient penalty strategy is proposed to measure whether the generated image is realistic and punish the redundant gradient produced by some of the generated images to stabilize the training process of the network.Finally,to further improve the diversity of the generated images,TDGAN uses the discriminator as a feature extractor and introduces multi-level image similarity loss to improve the feature similarity of the generated images and the corresponding real images at the levels of the foreground objects and the whole images,respectively.The three research contributions of this dissertation focus on text-to-scene image generation and respectively use the rule-based heuristic method and deep neural network-based method to study the technology from different perspectives.The proposed three research contributions have carried out sufficient experiments on relative datasets.Moreover,they have excellent performance on realism and diversity of the generated scene images and semantic consistency between the generated scene images and input text. |
