Research Of Head Pose Estimation Based On Fusion Of Depth Image And Color Image

With the popularity of information technology, Internet and Internet of Things become an absolutely necessary platform of human for daily life and work. The natural human-computer interactive pattern, which can interact with human, is attracting more and more researchers, and automatically recognizing the user’s action is an important research foundation of the area of intelligent natural human-computer interaction. As an important part of human action, not only we can obtain large amount of useful information from head only, but also it’s important to facial expression analysis and face verification and other related technology applications. However, the difficulties caused by head rotation and collecting environment becomes a key issue of the above application in practice, it’s still a hot and difficult issue for head pose estimation with rotation, lighting variation, so research of head pose estimation algorithm has important meanings.To improve the accuracy, the paper propose to fuse the easy detected face feature points in color image and depth image respectively, followed by feature calculation, finally learning the mapping relationship between the features and head posture. In the research, considering the difficulty of samples collecting and manual labeling in the application of machine learning algorithm, especially facing with the collection object of head and face in video and image, the paper proposes a kind of semi-supervised sequential online extreme learning machine(SOS-ELM). Experimental results show that feature points positioning is accurate and robust, and the learning algorithm is fast and has good generalization performance. The entire algorithm can estimate the head pose precisely, which will provide good technical support for subsequent application to human behavior analysis. The main research work is as follows:1. The paper designs a gray scale image conversion method based on YCbCr color space for eye detection. In the gray scale image converted from YCbCr color space, it is more accurate and robust to locate the eye based on Haar-like features.2. The paper proposes an improved Haar-like feature extraction method based on depth image. Using the rich information of depth image, the improved algorithms can extract spatial rotation invariant features, which is more suitable for head pose estimation than original Haar-like feature.3. The paper designs a facial feature points fusion method based on coordinate transformation. For the research object of this paper, feature-level fusion strategy is more suitable than the pixel level fusion and decision level fusion in terms of speed and accuracy. Especially after a simple coordinate transformation, the feature points from gray scale image and depth image can be integrated quickly and accurately.4. The paper proposes a semi-supervised online sequential extreme learning machine method(SOS-ELM). This method effectively solves the problem of labeled-samples are difficult to obtain, which is time-consuming and energy-wasting in practical. It make full use of the unlabeled samples, which is readily available, to get a model with good classification accuracy and generalization performance. In addition, the character of online learning makes it possible to complete the weight updating only by calculating the newly arrived samples, which saving computation and memory. Therefore, the algorithm is outstanding in training speed and learning performance.For all of the studies above, we have been tested them by experiments, and the experimental results works well and meet the desired demand. In particular, for semi-supervised online sequential extreme learning method, this paper carried out on a number of benchmark databases for experimental testing, including regression, binary and multi- classification database, we compare the proposed algorithm with SS-ELM and OS-ELM in all the databases above-mentioned, and the results verified the good performance of our proposed algorithm further.