Research On Depth Image And Point Cloud Completion Algorithm Based On Deep Learning And Video Stream

With the wide application of depth sensor,the processing algorithm of depth image and point cloud has begun to develop rapidly.3D image can represent 3D spatial information.Compared with 2D color image,it provides a new direction for many computer vision related research.However,3D images are often missing and damaged due to the problems of environment and acquisition equipment.The purpose of this paper is to use depth neural network in two kinds of three-dimensional images,depth image and point cloud,only using the incomplete image as input to infer the complete and reasonable three-dimensional image.In addition,the position and pose changes of depth camera are calculated in real time in the depth video stream,and the depth information from different perspectives is mapped to a single depth image to complete the repair of a single depth image.In this paper,firstly,for a single defect depth image,we use the two-dimensional image completion algorithm which has achieved good results in RGB color images,using the encoder-decoder network structure as the algorithm backbone network,replacing the traditional convolution layer with gated convolution as the image completion network.The calculation of surface normal in depth image is introduced to compare the effect of depth image restoration before and after adding surface normal.In the process of experiment,random line defect,edge position defect and large continuous defect are artificially made.In the case of mixed defects of various forms,the effect of image repair network is verified.Then the research of this paper is carried out in the point cloud.Two different point cloud encoders,Point Net and Point Net++ and two different point cloud decoders,Folding Net and PCN,are combined into four different networks.The incomplete point cloud is taken as the input to make the network output the missing part of the point cloud.The experimental part is carried out in modelnet40 data set,and the data set is artificially divided into residual part and patch part.Four kinds of network are trained to repair the point cloud.By comparing the repair effect of different networks in the test data set,the repair effect,performance and speed of different encoders and decoders are verified.Finally,aiming at the holes in the depth image caused by the defects of the depth image sensor,a computer with GPU is used to connect the depth camera to form a hardware platform,and a single depth image restoration system based on video stream is developed to observe the static target scene from multiple perspectives.Through GPU parallel computing acceleration,the depth image is obtained in real time and the pose change of depth camera is calculated.The depth information obtained from multiple perspectives is mapped to a single depth image through pose change,and the holes generated in the depth image acquisition process are filled.