Study On Hole Filling Method Based On Single Depth Image Information

With the development of the information technology,depth images are increasingly used in fields such as object detection,behavior recognition and scene modeling.Due to the defects of the 3D imaging system and external interference,the collected depth images often contain holes,and the hole is one of the main factors that limit the practical application of depth images.Currently,most methods for depth image hole filling are based on color image guidance.Complex acquisition equipment and the alignment accuracy between color images and depth images limit the application of such methods to a certain extent.The thesis focuses on the study of hole filling methods based on single depth image information,and the study results are as follows:(1)On the basis of studying the existing depth image inpainting methods,combining with sparse representation theory,a hole filling method using single depth image information is proposed.Firstly,the damaged image patch with the highest priority was selected as the image region to be restored.Then we created a group of similar image patches and used it as a inpainting dictionary.The sparse representation vector of the damaged image patch was solved by the orthogonal matching pursuit algorithm,and the damaged image patch was reconstructed by the linear representation of dictionary atoms.Experimental results showed that the algorithm can effectively fill small and medium-sized holes in depth image.(2)In order to solve the problem that the hole filling method based on sparse representation,a hole filling method based on U-Net and single depth image information was proposed.U-Net learned features from single depth image to infer the depth information in the hole.Finally,we combined the hole part of the network output with the original image to get the inpainting result.The experimental results showed that the hole filling results obtained by this method are similar to the optimal results of various hole filling methods combining color image information,under the condition of only learning features from single depth image.(3)To reduce the influence of invalid information inside the hole on CNN feature extraction,a hole filling method based on gated convolution U-Net and single depth image information was proposed.The gated convolution U-Net learned features from single depth image and effectively filtered out the invalid features inside the hole.It paid more attention to the effective features outside the hole to achieve more accurate feature learning and more accurate hole filling.The experimental results showed that the method can effectively reduce the influence of invalid features and achieve better performance of depth image hole filling than the U-Net based on traditional convolution.(4)For further improving the hole filling performance of deep learning methods,a hole filling method based on single depth image information and high-level edge structure information of depth image was proposed.We designed a two-stage network model.The network can complete edge structure inference and depth image hole filling in stages so that further improve the performance of depth image hole filling.From the perspective of qualitative analysis,it can be seen that the edge structure information can improve the performance of hole filling.However,due to the limitation of the training dataset size and the inaccurate edge inference results,the inpainting results of some depth images were affected to a certain extent.The thesis was dedicated to solving the problem of hole filling only using single depth image information.The design of hole filling method for depth image was studied from two aspects of sparse representation and deep learning.After testing,algorithms in the thesis had effectively improved the quality of depth image hole filling,and we will enhance their practical application value on the basis of further research.