On The Theory And Applications Of Feature Learning From High Dimensional Data

With the rapid development of information acquisition and transmission of information technologies, more and more massive amounts of data are to be processed,such as images, videos, webpage documents and audio data. Most of these data are of complex structure, particularly with high dimensions and redundant information.Therefore, how to analyse and deal with these data to get the intrinsic structure, and thus improve the performance of subsequent learning, decrease the time and computational complexity with the extracted data feature is a big challenge for researchers.To address this challenge, starting from the point of view of feature learning, this dissertation involves some basic knowledge of linear feature learning, nonlinear feature learning, and linear approximation to learning nonlinear, in addition, this dissertation covers the differences between local structure learning and global structure learning, and the extension from the vector-based classic machine learning methods to the tensor-based ones; Especially, this dissertation focuses on the local structure feature learning of the complex data, and the ways to explore the intrinsic structure and discriminant feature structure of the complex data which can largely condition the success of any subsequent machine learning endeavor. To achieve these,we propose several novel feature learning methods. Specifically, the main contributions of this dissertation are summarized as follows.1. Concept Factorization(CF) is a purely unsupervised feature learning method and it cannot use the priori knowledge to guide the process of clustering. To address this issue, we propose a Semi-Supervised Concept Factorization(SSCF)method. Based on the standard CF, SSCF is proposed by incorporating the pairwise constraints into CF framework as the reward and penalty terms, which guarantees that the data points belonging to a cluster in the original space are still in the same cluster in the transformed space. SSCF provides a dynamic penalty mechanism which better accounts for the intra-cluster variance, i.e., allowing dissimilar data points with the same cluster label to be mapped farther than similar ones. In this way, we can learn more reasonable clustering structures in the representation space. To infer the document clusters, an iterative algorithm is proposed to perform the term-document matrix factorization, and the convergence of SSCF is proved in the main theorems. To validate our proposed method, extensive experiments are conducted on publicly available data sets, our experiments show the superior performance of SSCF for document clustering.2. Although semisupervised CF methods are very popular, their performance may be degraded for their neglecting the local geometric structure of the unlabelled data. To address this issue, we propose a Constrained Neighborhood Preserving Concept Factorization(CNPCF). CNPCF uses pairwise constraints and information related to a local invariant for achieving better learning performance,where the local invariant is based on a generalized meaning of closeness which includes not only spatially close points in the geometric space but also points directly connected by must-links. To encode such information, we use a p-nearest neighbor graph(which is mainly for capturing the local geometric structure of the dataset) and a membership graph(which preserves the similarity of those must-link constrained pairs). To take into consideration of all such information,a carefully designed objective function is used for the factorization process. Particularly, a penalty term is added to the objective function for each violation of the pairwise constraints. To preserve local invariant, a term corresponding to each of the p-nearest neighbor graph and the membership graph is added to the objective function. To optimize the objective function, we develop an iterative scheme, and show its convergence. The achieved data features using CNPCF can better represent the original data than those using the state-of-the-arts methods.3. Locality Preserving Projections(LPP) is a classic unsupervised manifold learning method and it cannot take advantage of the priori information to improve its performance. To address this issue, we propose a novel method called Locality Consistent Discriminant Analysis(LCDA). LCDA incorpoates two graphs(the intraclass graph and interclass graph) into the LPP framework, and by designing the objective function elabrately, LCDA makes the data points in the same class more compact and the data points in the different classes farther. LCDA method improves the permance by preserving class separability as well as the intrinsic geometry structure of the data points. Experimental results on some public data sets show the effectiveness of the proposed LCDA method comparing with the state-of-the-arts techniques for face recognition.4. We propose a tensor tree feature learning framework, disclose the relationship between the framework and the classic tensor factorization methods such as Tucker factorization and CP factorization, and present a novel method based on the framework called Neighborhood-embedded Tensor Learning(NTL). The data are now becoming more diverse, more massive and more high-order, which makes tensor attract extensive attention due to its effectiveness in reprentation and anlysis of complex data. Despite the important of tensor, there is no complete tensor theory system. Therefore, we try proposing Tenser Tree Learning(TTL) methods and constructiong a tensor tree feature learning framework, which enriches and develops the tensor learning research. Furtherly, based on Discriminant Neighborhood Embedding(DNE), we propose a new method called NTL. DNE is a famous feature learning method, however, it has some limitations: it neglects the role of the points out of the neighborhood of a data point, and it might lead to the so-called “curse of dimensionality”and “small sample size problem”due to its vector-based character. The proposed NTL not only inherits the power of DNE but also has the advantage of tensor-based methods. By designing an objective function elaborately, NTL maps the data into a low dimension subspace where the data in the same class are more compact and the data in the different classes are more separable. Such design of the objective function improves the discriminant power of the method. Experimental results on the three public data base(ORL, PIE and COIL20) demonstrate the effectiveness of NTL methods comparing to the state-of-the-arts techniques.