Implementation Of3D Camera Based Depth Image Processing On FPGAs

3D video/image processing is one of the hottest research topics. Currently, many researchers of China and aboard are putting their focus on how to get3D video/images with high resolution and quality in a fast computation fashion. There are numerous discussion and papers about this topic right now. However, there is no uniform processing standard for this area so far. In current situation,3D processing method could be mainly divided into three directions:the first one is stereo matching based on obtaining object information from cameras in multi-direction; the second one is to reconstruct existed images for getting their3D information; the third one is to enhance disparity frames from current3D cameras. For stereo matching method, there are many researcher have proposed their algorithms, and some of them got acceptable processing results. However, the shortage of this method is the way to get video/image from real world, because the way of putting cameras (the location, angle etc.) will affect the speed and precision of the algorithm. For image reconstruction based method, it is more suitable to be applied for recovery of historical objects. Additionally, this method cost intensive computation, and thus makes it almost impossible to implement in real time processing system even in hardware design. Compared with the two previous methods, the disparity camera based method is relative newer in recent years. This processing strategy is to get complete3D video/image by enhance both quality and resolution of disparity images. This thesis focuses on very strategy and develops deep discussion. I believe that, with the wide application of disparity camera, by using enhancement for disparity images to get high resolution and high quality will become more and more popular and have important research value.This thesis focuses on how to enhance the disparity images and to get final3D video/image with high quality and resolution in real time. We give a deep insight on the way of getting final disparity images in both software and hardware. Our work could be extracted into two parts:software algorithm research and hardware implementation.First, our main contributions in software algorithm include the following: (1) Research of global interpolation method based on weighted functions under multi-scale situationWe adopted multi-scale strategy when applying interpolation computation for disparity images. This thesis provides a binary module matrix with different interpolation levels. In this way, the output data could have smoother view and also could decrease noise. Meanwhile this method could make interpolation computation easier to determine which pixel points need to be interpolated while which do not. By employing this strategy, the processing results of disparity images have less mosaics effects, thus could make the images closer to real world. Last not the least, this method could improve the speed of the system.(2) Simpler computing scheme for temporal consistencyAfter the work of (1), our system takes the coarse processed disparity images data, which together with later data from optical flow based computation, as the input of post processing stage. In this stage, we proposed a simplified way to calculate temporal consistency compared with original complex algorithm. Under this way, the error rate of final processing data in an acceptable range and the images thus could still keep high quality. Because of reducing the burden computation, the system has faster speed.Second, we have the following contributions in hardware implementation:(1) We proposed a combined storage way in odd and even fashion to capture raw data from outside camera. This method could extract R, G, B vectors separately in a fast way, and then generate the necessary video format for next stages.(2) We proposed a row based Gaussian filter computation architecture. In this architecture, the system takes the pre-stored the coefficients of Gaussian module, which together with the data need to be processed, as the inputs of convolution unit, and then execute the convolution calculation. Meanwhile we applied shift fashion to replace dividers and then could make the system faster.(3) We proposed partial parallel architecture for combined unit of multi-scale based interpolation and optical flow computation. In this architecture, system processes the nth level of interpolation and the (n-1)th level of optical flow computation in a parallel way. This design greatly improves the speed of system. For the separate interpolation and optical flow computation unit, the system employs cascade way for the implementation. (4) We proposed a hierarchy based architecture for optical flow computation unit. This architecture contains one fast bilinear interpolation unit, one gradient generation unit, one least square matrix computation unit and one velocity computation unit. The strategy of hardware architecture also employs partial parallel fashion to improve system speed. That is, the nth level of bilinear interpolation and the (n-1)th level of gradient convolution unit could be performed concurrently. Similarly, for the bilinear interpolation unit, the computation performs in a cascade way.(5) We proposed a parallel architecture for post processing stage, namely the temporal consistency computation stage. This hardware architecture could further improves the performance of the system, and makes convolution of the previous frame of disparity image and current frame of disparity image could be parallelized with convolution of current frame of disparity image and current motion estimation matrix based on optical flow. Therefore, both performance and speed of the system could be enhanced this way.