Multi-sensor Fusion For 3D Scene Perception

Machines can replace human being to do some dangerous,repetitive work or obtain accurate information on 3D world once they have the capability of 3D perception.They acquire scene information by sensors which may be limited by ranges and random noise.Fusing data from multiple sensors can not only improve the system stability,but also obtain more accurate and completed information.In this thesis,we explore fusing multiple sensors to perceive the 3D world in two steps.The first step is to combine an image sensor and an acceleration sensor in the smart mobile device for measuring the real distance of the observed 3D scene,including depth,ground distance,object size and so on.This method can capture accurate information conveniently,however,it is limited to the type and number of sensors ony the smart mobile device,and limited range and scene information.Therefore,we increase the sensor number in the second step with additional depth sensors.This method can reconstruct a complete model of dynamic scene in real time and thus be applicable for games,virtual reality,movie effects and so on.Accordingly,the main work of this thesis is as follows:1.Single-image based distance measurement using the embedded sensors of a smart mobile device.Existing smart mobile oriented measurement work either requires multiple image or is only valid for only one distance type.We propose a single-image based method which cleverly measure various types of distance from single image captured by a smart mobile device.The embedded accelerometer is used to determine the view orientation of the device.Consequently,pixels can be back-projected to the ground,thanks to the efficient calibration method using two known distances.Then the distance in pixel is transformed to a real distance in centimeter with a linear model parameterized by the magnification ratio.Various types of distance specified in the image can be computed accordingly.Experimental results demonstrate the effectiveness of the proposed method.2.A two-level volume based fast multi-depth images integration method.This method addresses the shortcomings of the current volume fusion algorithms which are low speed and inconvenient by dividing the 3D space into two layers of volume.Multiple depth images from sensors can be integrated quickly with a high fusion quality by optimizing patterns to access memory and specially designed data structures.First,inspired by sparse representation of octrees,an efficient data structure and a fast fusion algorithm for integrating depth images in parallel in GPU are designed.The data structure of the root layer and leaf layer are efficiently coded and compressed to reduce the memory occupancy and access times.Second,an efficiently circular queue-based memory pool is implemented in GPU to reduce frequent memory requests to the operating system.Thirdly,the marching cubes algorithm is extended to extract isosurface from the two-level volume,which greatly reduces the number of memory accesses.Compared with the original algorithm,the time consumed is significantly reduced.Finally,parallel strategies based on the current GPU architecture are designed for the two-level volume fusion algorithm.3.A real-time dynamic scene 3D reconstruction algorithm based on multiple sensor.Fusion of all the depth data may still contain a large number of holes and noise because of the obstruction and inaccuracy of sensors.In addition,non-rigid motion of dynamic scene also increases the fusion difficulty.Plus,the reconstructed mesh stream is huge and far exceeding the internet transmission capabilities.We apply key frames and non-rigid registration techniques to denoise,fuse,and compress data.Firstly,an iterative algorithm with projected nonlinear constraint term is designed to register two meshes.The nonlinear term of the objective function is extracted and converted into a linear constraint problem by projection.The original nonlinear least squares problem is transformed to the alternation of the linear least-squares problem and projection.The solution of projection operation is closed and can be quickly obtained through parallel polar decomposition.Compared to directly solving original problem in non-rigid registration,the algorithm is faster and more robust than existing methods.Then the two models after registration are fusion in volumes.Finally,we compress the transformation parameters of key-frames and non-key-frames,greatly reducing the amount of data that needs to be transmitted.