Research On Hardware Architecture For Real-time Visual Feature Detection And Matching

As the rapid development of science and technology, with its specific and intuitive features, image becomes the primary mean to access information, and is widely used in data representation, computer vision, pattern recognition, medical imaging, remote sensing image processing, image retrieval, navigation and guidance, and3D reconstruction. Some of these applications demand to build correspondences between two or more images, which may be from different sensor, times, depth, and viewpoint, in real time. However, due to its computational complexity, its pure software implementation without using special hardware is far from satisfactory in its performance for real-time applications. This paper is focus on designing independent, universal, low power, small size, and real-time visual feature detection and matching hardware architecture.Firstly, based on a large number of analysis and comparison, we found that the SIFT algorithm is one of the most robust feature detection and description algorithm, its significant advantage over other methods is that the SIFT feature is invariant to image translation, scaling and rotation, while at the same time quite robust to illumination change and affine or projection. However, it is known that it is very difficult, if not impossible, to achieve software-based real time computing due to its computational complexity. We proposed a very efficient hardware architecture based on FPGA associated with high performance DSP to detect and describe SIFT feature in real time. We optimize the2D Gaussian filter according to the fact that the Gaussian kernel is separable and symmetrical. We also proposed a full parallel/pipeline feature detection architecture and determine the pixel width by experimentation. Due to these optimizations, the proposed architecture can reduce the utilization of the FPGA resources without performance degradation. It only takes10ms to detect SIFT feature from320x240image, and takes merely about80us per feature at feature description. In general, the number of SIFT feature per image is about200in our applications, hence, the proposed architecture can extract SIFT with50fps. Although the proposed FPGA+DSP architecture can detect and describe SIFT feature in real-time. However, it requires a circuit board with two chips to detect and describe SIFT feature. We proposed a very efficient visual feature detection and matching system based on single FPGA chip. It reduces the FPGA resources utilization by analyzing the pixel throughout and computational requirement, optimizing the cascade architecture of the2D Gaussian filters, and exploring the parallel/pipeline of BRIEF feature extraction. It can detect SIFT features from1280x720images, extract BRIEF descriptors for detected features, and match BRIEF features in consecutive frames at a frame rate of60fps. This system is able to comfortably achieve real-time feature detection and matching computation for image sequences. Our extensive experiments shown that the processing speed has been greatly improved, while keeping quite the same detection and matching accuracy. As the proposed system is a completely stand-alone on-chip system and can be driven by video cameras directly, it can be embedded into many real-time systems, such as smart cameras, visual based robotics systems, and real-time image alignment systems.Finally, we utilize the proposed visual feature detection and matching architecture in two applications. Firstly, utilize the FPGA+DSP based SIFT feature detection and description architecture to improve the performance of the horizontal velocity estimation for lunar by speed up the most time consuming part. Secondly, utilize the proposed FPGA based feature detection and matching architecture to speed up the disparity estimation for uncalibrate stereo cameras by providing real time and robust correspondence. It doesn’t need the internal parameters and the geometric information between cameras is known.In summary, we proposed two hardware architectures for real time visual feature detection and matching. These two architectures can meet the power consumption, size and real time performance constraints of real life computer vision system. It is theoretical significance and application valuable in the navigation guidance system, robot stereo vision system, target tracking, and3D reconstruction.