Person Removal And Inpainting For Images Based On Generative Adversarial Networks

With the development of society and the improvement of people's living standard,photography has become an indispensable part of life.However,unrelated people are often mixed in when taking landscape photos,personal photos or group photos,which affects the photographer's shooting experience.It is necessary to remove the unrelated people and inpaint the background,so that the picture looks natural without a sense of violation.Generative Adversarial Nets(GAN)is a deep learning model that draws on game theory ideas to achieve good output through mutual game learning between the generative model and the discriminative model.In this thesis,generative adversarial networks are employed to study image inpainting,which achieves person removal and background inpainting combining semantic segmentation.The main contents of this thesis are as follows:1.Build an image dataset for whole person removal.The dataset contains two categories of scenes(single-person vs multiplayer)with a total of 16,000 images.All the images are manually selected from Baidu,Google,and the places2 scene dataset.The single-person scene category can be further divides into single background and complex background,each contains 3000 images.The multiplayer scene category can be further divides into four sub-categories: dispersed person&simple background,closed person&simple background,dispersed person&complex background,closed person & complex background,each contains 2500 images.Five hundred images are selected randomly from above 6 sub-categories and a total of 3000 images are collected as the semantic segmentation dataset.The images in which are manually labeled with Labelme(a labeling tool)and transformed so as to meet the requirements of semantic segmentation.2.Employ the semantic segmentation technology and generative adversarial network to achieve person removal and background inpainting for images.The semantic segmentation network,which is designed based on U-net structure,is used to automatically generate masks for persons.PR-GAN is designed for image inpainting by drawing on the idea of generative adversarial network.The network contains a discriminator and two generators,which perform rough inpainting and fine inpainting,respectively.Fine-tuning is adopted to train the model.The experimental results show that,(1)in the single-person scenarios,the PSNR and SSIM index of the proposed model reach 19.014 and 0.7347,which is increased by 0.372 and 0.0085 compared to the pre-trained model,respectively;the images with better removal effects account for 66% of the total number of generated images,which is 9% higher than the pre-trained model.(2)In multiplayer scenarios,the PSNR and SSIM of the proposed model reach 19.637 and 0.7309,respectively,which are increased by 0.184 and 0.0018 compared to the pre-trained model;the images with better removal effects account for 58% of the total number of generated images,which is 5.5% higher than the pre-trained model.3.By drawing on the attention mechanism,an improved PR-GAN network is designed to achieve person removal and background inpainting for images.The improved PR-GAN adds an attention layer between the 6th and 7th convolutional layers of the second branch of the second generator,and employ feature maps with attention to further improve the performance of the model.The experimental results show that,(1)in the single-person scenarios,the PSNR and SSIM index of the proposed model reach 19.468 and 0.7443,which is increased by 0.454 and 0.0096 compared to the fine tuning model,respectively;the images with better removal effects account for 74% of the total number of generated images,which is 8% higher than the fine tuning model.(2)In multiplayer scenarios,the PSNR and SSIM of the proposed model reach 19.696 and 0.7420,respectively,which are increased by 0.059 and 0.0111 compared to the fine tuning model;the images with better removal effects account for 65.5% of the total number of generated images,which is 7.5% higher than the fine tuning model.