Research On Virtual Viewpoint Generation Algorithm Based On Spatio-temporal Joint Repair

With the characteristics of interaction,immersion and real-time,free view video has become the development direction of the next generation of digital video technology.It has a huge market demand and application scenarios in medical research,video surveillance,video entertainment and other aspects.Because of the high cost and large amount of data in the traditional method of obtaining free view video,the virtual view rendering technology which uses the known view to generate any view has become the first choice to generate free view video.Depth image based rendering(DIBR)has become the key technology of virtual view rendering because of its advantages such as fast rendering speed,low network transmission burden and unlimited virtual view.However,there are a lot of cracks,artifacts and large area of continuous holes in the virtual view generated by DIBR technology,which seriously reduces the quality of the virtual view.At present,the existing solutions have good repair effect on cracks and artifacts,but for the repair of large-area holes,they are not only lack of semantic sense and block effect,but also limited to single frame images,ignoring the temporal continuity of foreground motion information between video frames and the redundant information between video frames,which leads to the repair of virtual view when playing Flicker occurs.To solve these problems,this paper proposes a method based on the space-time domain to jointly repair the holes.The main work completed is as follows:(1)A method of hole repair in Edge-PUnet network based on spatial domain.Firstly,the method based on pixel assignment and expansion is used to remove the cracks and artifacts,so as to eliminate the influence on holes repair.Then,an edge detector is designed on the PUnet network for the part where the holes usually appears at the junction of foreground and background in the virtual view,and the edge is rich in texture and structure in the image.Edge detector is mainly used to detect the boundary between foreground and background in data set,so that PUnet network can extract the structure and texture features of image edge.Finally,the model after training is used to repair the holes in the virtual view after removing the cracks and artifacts.The experimental results show that this method can effectively solve the problem of block effect and semantic loss caused by the traditional image restoration method.At the same time,compared with the PUnet network,the edge of the repaired holes area is more clear and has good structural information.(2)A method of hole repair in optical flow guided pixel propagation based on time domain.In this method,the foreground region and background region in the virtual view video hole are repaired respectively.For the restoration of the foreground region,the idea of stream guided pixel propagation is introduced,which is to extract the spatiotemporal optical flow field of foreground motion information between video frames,and then use the optical flow to guide the propagation of foreground pixels in video frames.For the background area and the foreground missing area which can not connect to any other pixels in the frame,the Edge-PUnet network based on the space domain is used to repair.The experimental results show that this method not only improves the quality of single frame virtual view,but also effectively guarantees the continuity of the repaired image in time,and the repaired image transitions naturally when playing.