Study On Mural Inpainting Based On Structure-guided Fourier Convolution Generative Adversarial Networks

The Dunhuang Mural,located in the Mogao Caves in Dunhuang,is a world cultural heritage,It is a vast.historic and richly endowed recording of the glorious and colourful Chinese culture.However,affected by both natural and human factors,Dunhuang Mural have suffered from mild or severe types of damage such as weathering,mildew and smouldering.Due to the large size,irregularity and the complexity of the cave environment,the damage to Dunhuang Mural poses a great challenge for manual inpainting.Therefore,digitizing Dunhuang murals for storage and inpainting is currently the most efficient and effective way to do so.With the rapid development and application of deep learning technology in image,deep learning-based image inpainting algorithms are gradually replacing traditional defuse-based and patch-match algorithms with effectiveness and speed.However,for images with complex structures and rich textures such as murals,these methods still have the problems of structural errors,texture blurring,and content hazy due to the lack of a priori information.To overcome these difficulties,this paper proposes a two-stage mural inpainting algorithm based on structure-guidance and fourier convolution generation adversarial network,which processes the structure information and texture information separately,meanwhile the structure of the missing region of the image can also be predicted by the algorithm;the texture features are extracted by frequency-domain convolution to obtain a larger perceptual field in the high-dimensional space;the fusion of the feature information from both aspects is used for inpainting,leading to a higher inpainting quality.This paper’s main contributions can be summarized as the following three aspects.(1)Proposed a mural structure prediction network based on an improved Transformer architecture.Using the edge detection algorithm and wireframe extraction algorithm to obtain the edge information and wireframe information of the image separately,the Transformer is then used to predict the structure information of the missing regions in order to obtain the real structure closer to the original image.Also,a Dunhuang Mural dataset suitable for deep learning training is established.After collecting,classifying and pre-processing,a dataset of 15,948 mural images and 3,000 masks simulating mildew damage has been produced.(2)Proposed generative adversarial network based on Fourier convolution.In order to have a better view of the global context,this paper integrates the guidance of structural information for image inpainting,as well as using Fourier convolution to extract high-dimensional features.Then,by applying global and local feature extraction,the inpainting result can be closer to the surrounding area.(3)Designed a Dunhuang mural inpainting system based on the algorithm of this paper.A user-friendly one-click mural inpaitng system is designed to make the algorithm easier and more efficient to use,which allows user inpainting any specified area manually.This system includes functions such as image upload,mask drawing,image restoration and so on.