Real-time 3D Scene Reconstruction With Mobile Devices

3D reconstruction is the foundation of stereovision,and it is a core problem in computer vision.Rencently,mobile devices have been equipped with various sensors and their data acquisition and processing capabilities are continuously enhanced,which makes the mobile device become a new and convenient tool for fast 3D dada acquisition,3D scene understanding and effective information acquisition.Compared to the computer,the quality of the mobile device still needs to be improved.The main challenges for 3D reconstruction with the mobile device include dealing with the noisy and low-frequency depth image,and managing limited computational and memory resources.Therefore,the filtering and upsampling,which is also known as super resolution reconstruction,of the original depth image are investigated in this paper.The aim is to improve the quality of 3D reconstruction by improving the resolution of the depth image.The device used is Lenovo PHAB2 Pro,a cell phone of Google Tango.First,the data acquisition and preprocessing are introduced,including the technical parameters of the device,the principle of depth measurement and the alignment of depth data and color data.Then,the source and characteristics of the noise are analyzed in view of the strong noise and low resolution of the original depth image.The nearest interpolation(Neareat),joint bilateral upsampling(JBU),discontinuity adaptive depth upsampling(DADU)and an improved algorithm are used to process the original depth data.The upsampling results of the 2D depth images and the 3D points cloud obtained by different algorithms are compared.The results show that the strong noise region of the depth image can be processed effectively with the proposed algorithm and the boundaries can be preserved through introducing a color constraint in the local minimum filter.Finally,a 3D reconstruction system is developed for Lenovo PHAB2 Pro based on the proposed algorithm,and the process includes data acquisition and processing,upsampling of the depth image,acquiring single frame color point cloud,reading camera poses from VIO module,TSDF fusion by voxel block hashing,mesh generation by Marching Cubes and Open GL ES rendering.Every modules of the reconstruction system are described in detail in this paper and implemented on Lenovo PHAB2 Pro.The proposed 3D reconstruction system can be employed to reconstruct objects and3 D scenes in real time on a mobile phone.In addition to the GPU processing of themesh rendering module,the calculation of the other module is performed on the CPU.Because of the limited computing power of the mobile device and the large computational complexity of the algorithm,the real-time performance is reduced.It is necessary to use GPU and other acceleration algorithms to achieve further optimization.