Research On Machine Learning Based Virtual View Rendering

With the constant improvement of people's living standard and the rapid development of multimedia video technology,traditional 2D video can no longer meet the needs of users for stereoscopic visual effects,so Free Viewpoint Video system(FVV)came into being.FVV system includes the process of video content capturing,encoding,transmission,decoding,rendering and display.It allows users to freely choose any viewpoint,but if the system capture all the content of each viewpoint,it will bring about the problem of excessive data.Therefore,the FVV system adopts the depth image based rendering method(DIBR)to solve the problem of large data caused by multi-view viewing requirements.It only needs to capture some limited viewpoint information at the encoding end,and then it can be rendered into virtual view at any position,which greatly saves the transmission bandwidth of network.Due to the quality of depth map and rendering technology,the rendered image will inevitably exist the problem of hole filling.If the holes are filled by mistakes,there will be many distortions such as artifacts and object deformations in the virtual view image,which seriously affects user's subjective visual perception.In recent years,machine learning technology has achieved excellent performance than traditional algorithms in areas such as image denoising,image recognition and image inpainting.Therefore,in view of the problem of rendering holes in the DIBR technology in the FVV system,this paper proposes to use machine learning to solve the problem of hole filling in virtual view rendering.(1)Aiming at the problem that the virtual view still exists poor visual perception after filling the hole by traditional algorithm,this paper proposes a virtual view hole filling algorithm based on convolutional neural network.The algorithm includes image preprocessing,feature extraction and hole filling.Firstly,the reference view images are preprocessed,which uses the DIBR technology to obtain merged virtual view images and the hole masks.Secondly,multi-dimensional features are extracted from virtual view image by convolutional neural networks and the structure similarity loss function is derived from the hole area to optimize the network.Finally,the hole in virtual view image is filled by the extracted features.Experimental results indicate that the algorithm can effectively preserve the sharp edges of the foreground and background in the virtual view image,while effectively fills the hole area.Whether it is subjective visual perception or objective quality evaluation,our algorithm has achieved satisfactory results.(2)In order to further improve the quality of virtual view rendering,this paper proposes a virtual view hole filling algorithm based on edge learning.Considering the number of scenes in the existing rendering sequence is limited,this paper constructs a natural image data set with edge hole.First,download 50,000 natural images.Then,extract the edge mask that conforms to the rules of holes in the virtual view rendering,and add the edge mask to the original natural image to obtain a constructed data set of holes with different sizes.This paper uses the proposed hole filing network to train on the constructed data set,and taking tests on the constructed data set and render scene data.The subjective experimental results show that the network model trained on the large-scale hole constructed data set has a good hole filling effect for the above two types of images.At the same time,it also verifies the rationality of the constructed data set and the effectiveness of the proposed network model.In order to accurately learn the rules of hole filling in the virtual view image,the hole filling network is used to train the data set containing the combination of the constructed data set and the render scene data set separately,and finally taking tests for the virtual view hole image.The experimental results show that the proposed algorithm can effectively fill the holes of the virtual view image and improve the visual perception quality of the render image.