Researches Of Vision Based System For Extracting Driver’s Viewpoint

Advanced driving assistance system(ADAS) will be extensively used in future vehicle, and has a wide application prospect. In this dissertation, conbining with computer vision technical, we proposed a driving assistance system to extract the driver’s view point. So the system can give an alarm when there are potential dangers. Around this driving assistance system, several key technicals have been researched. It can be devided into three parts: how to calibrate RGB-D camera system and find corresponding pixels in depth image for the pixels of color image; how to estimate the relative pose of two in-vehicle cameras with non-overlapping field of view(FOV); how to recognize human actions after one shot learning.In our driving assistance system, RGB-D camera will be used. In order to improve the accuracy of depth measurement and improve the performance of this system, the RGB-D camera should be calibrated. The perspective projection model of normal cameras is expanded for infrared camera. Then a camera system model for Kinect is builded, including color camera, infrared camera and depth camera. The traditional calibration method is used to calibrate the internal and external parameters for color camera and infrared camera. Next, using the model of depth camera, the depth parameters can be calibrated. Results of experiment show high accuracy. However, there is still a problem for RGB-D camera system. Because color image usually has more context information than depth image or infrared image, we would like to find interesting points in the color image and then mapping to the corresponding depth image or infrared image. But the current SDK of Kinect can only mapping pixels from depth image to color image. Aiming at this problem, we proposed an algorithm for mapping pixels from color image to depth image, where the Epipolar principle has been used. The effectiveness of this algorithm has been proved through experiments. The error is 0.4 pixels on average. The innovation points for this part are as following:A model is builded for RGB-D camera system to calibrate parameters of the camera system.Using the epipolar principle, an algorithm is proposed to map pixels directly from color image to depth image.For the vision based ADAS, several cameras are usually used together as one. There are usually no overlapping fields of view among such cameras. For using several cameras as one, the relative pose between any two cameras should be known. For example, in the proposed driving assistance system, one camera is used inside the vehicle to track driver’s sight direction; the other camera is used outside the vehicle to monitor the outside scene. To accurately extract the driver’s view point from the image of the outside camera, the sight direction in the coordinate system of the inside camera should be mapped to the coordinate system of the outside camera. To realize this, the relative pose of the two cameras is necessary. As the two cameras have no overlapping views, traditional calibration method can not be used. Aiming as this problem, we proposed using a laser pointer fixed on a calibration board. By connecting the two views of the two cameras, a coplanar constraint or a collinear constraint can be used to optimize the relative pose of the two cameras. Before that the relative pose between the laser pointer and the calibration board should be calibrated. It can be done using the ideal camera projection model and the unit direction principle of the laser ray. Through an error analysis, we come to a conclusion that, the error of estimated laser direction is under 0.36 degree, so this method is feasible. For the laser based method, two algorithms have been proposed: the coplanar constraint algorithm and the collinear constraint algorithm. Another calibration board will be placed in the view of the outside camera, when we use the collinear constraint algorithm. The two algorithms are compared with the state-of-the-art algorithm(mirror based method) through simulation and real-world experiments. The results of simulation and real-world experiments show that, collinear constraint algorithm outperforms the mirror based method in term of translation accuracy especially when the two cameras apart a long distance. The innovation points for this part are as following:Based on the ideal camera projection model and unit direction principle of laser ray, propose an algorithm for estimating relative pose between laser pointer and calibration board.With the laser pointer fixed on a calibration board, propose two algorithms for estimating relative pose of two cameras with non-overlapping views.Human action recognition is an important branch of computer vision, which can be used in human-computer interaction, game development, robot application. Also, it is an important part of advanced driving assistance system. For example, with action recognition the vehicle may judge the meaning of traffic policeman or pedestrain’s hand single, even the driver may control the vehicle using action. We propose a novel action representation. Take the action as a collection of key poses. Segment and recognize actions using the key poses. This method is especially suitable for pre-processing and giving a quick classifying. Comparing to the past methods, which learn based on a large number of training samples, our method uses only one training sample. The experiments show that, our method is suitable for classifying actions which are slightly different with each other. Because our method considers time characteristic and space characteristic at the same time. The innovation point for this part is as following:Try to represent an action with a collection of key poses. And this extremely simplified the representation and recognition of actions.