Research On Dimensionality Reduction And Its Application In Feature Extraction

Faced with the growing massive data, people increasingly rely on computers to intelli-gently get useful information from the data to solve problems. As an important technique of in-telligent data analysis, dimensionality reduction not only effectively reduces the computational complexity of the processing procedure, but also significantly improve the accuracy and validity of data analysis. Dimensionality reduction techniques are widely used in applications of pattern recognition and computer vision, which makes feature extraction become the key to solve re-lated problems. Despite the research for dimensionality reduction has made fruitful results, but the new characteristics of current data with high dimensionality and multi-modal features being new challenges to dimensionality reduction target. Driven by the requirements from applica-tions such as face image recognition, video analysis and multispectral remote sensing image processing, dimensionality reduction technique has made further development through improv-ing and perfecting the existing methods or exploring new theories and techniques. Standing on the reality of data analysis demand in current situation, this thesis studies some theoretical and algorithmic problems in dimensionality reduction for vector data and high-order data, as well as some practical problems of feature extraction in applications. We develop some new algo-rithms as regards it, and these algorithms are verified to be effective in the application of data visualization and face identification.Manifold embedding methods have advantages in discovering the underlying structure of data, thus it is currently a research focus in unsupervised dimensionality reduction for vec-tor data. However, manifold embedding methods cannot obtain explicit mapping between data space and feature space, which leads to the difficulties when one try to apply the methods to new data. To solve this problem, we develop a novel algorithm named manifold oriented stochastic neighbor projection(MSNP) is developed for unsupervised feature extraction. This algorithm is inspired by the stochastic neighbor embedding(SNE) algorithm and owns the characteristic that it uses an explicit linear projection from data space to feature space to approximate the nonlinear manifold mapping. Based on the analysis results about the deficiency of SNE al-gorithm, we achieves improvement and perfection on the following three aspects:(1) MSNP models the structure of manifold by stochastic neighbor distribution in the high-dimensional observation space with geodesic distance rather than Euclidean distance, improving the accu-racy of estimating the pairwise similarity of data. (2)In order to enhance the adaptability to data with variable Characteristics, MSNP used Cauchy distribution in low-dimensional feature space instead of the Student t distribution to calculate stochastic neighbor selection probability. (3) MSNP benefits from stochastic neighbor distribution preserving strategy and linear projec-tion manner to endow with nice properties, with a fast convergence speed given by the solution based on conjugate gradient operation. We evaluate the effectiveness of our MSNP method for dimensionality reduction, including the property of basis vector, convergence speed and the ability of feature extraction. The experimental results on ORL, Yale, AR, and PolyU databases demonstrate that MSNP can substantially enhance the quality of data visualization compared with many competitive manifold learning algorithms and improve the recognition accuracy in biometrics recognition task. This verifies that the proposed MSNP is an effective feature ex-traction method.Local linear discriminant analysis takes data structure information into account for unsu-pervised feature extraction of vector data, so that it obtains feature with more discriminative power than the global discriminative analysis method. We make a thorough study on the exist-ing local linear discriminative methods, then find that there exist multiple parameters in most of these methods and it is difficult to set the proper parameters. Aiming at this problem, we make exploration and research on the automation of local discriminative analysis for dimensional-ity reduction. We develop a novel algorithm named adaptive local discriminative analysis, in which there is only one parameter needed to be set. This algorithm performs discriminative analysis by using the difference model based on a novel local neighborhood, in which the local inter-class neighbors are determined automatically. We design the adaptive algorithm under the principle that the model parameter is automatically determined according to the local data distribution of inter-class neighbors and intra-class neighbors. The experimental results show that the proposed adaptive achieves higher recognition accuracy than the existing methods in most case, and in the special case of 2 samples available per class it gets comparable results. Considering that the adaptive algorithm need to set only one parameter, the results provide proofs that our algorithm is effective for supervised feature extraction.Several tensor based dimensionality reduction methods are proposed recently to reduce the dimension of high-order data, such as images and videos. Although some algorithms such as Tensor LPP and Tensor NPE consider the nonlinear structure of data, the manner of linear projection restricts the existing tensor methods on the ability of discovering nonlinear structure. In this paper, we develop a novel unsupervised dimension reduction algorithm for the purpose of exploring the nonlinear structure information from high-order data. We design the algorithm under the assumption of tensor manifold, and combine local rank-one tensor projection and global alignment strategy to obtain the embedding of tensor manifold. Besides, we provide a scheme of numerical interpolation to solve the out-of-sample problem. The experimental results of data visualization show that the proposed tensor manifold embedding algorithm can discover the underlying nonlinear structure of high-order data. And its effectiveness of feature extraction is proved by the experimental results of face recognition.For the supervised feature extraction task to high-order data, we propose a new tensor based algorithm, called local discriminant orthogonal rank-one projection(LDOROTP). The goal of LDOROTP is to learn a compact feature for images meanwhile endow the feature with prominent discriminative ability. LDOROTP achieves the goal through a serial of rank-one tensor projections with orthogonal constraints. To seek the optimal projections, LDOROTP carries out local discriminant analysis, but differs from the previous works on two aspects: (1)the local neighborhood consists of all the samples of the same class and partial local sam-ples from different classes; (2)a novel weighting function is designed to encode the local dis-criminant information. The criterion of LDOROTP is built on the trace differences of matrices rather than the trace ratio, so the awkward problem of singular matrix do not emerges. Besides, LDOROTP benefits from an efficient and stable iterative scheme of solution and a data pre-processing called GLOCAL tensor representation. LDOROTP is evaluated on face recognition application on two benchmark databases:Yale and PIE, and compared with several popular projection techniques. Experimental results suggest that the proposed LDOROTP provides a supervised image feature extraction approach of powerful pattern revealing capability.