Small-sample Learning For High-dimensional Problems

Small sample learning (SSL) is a hot topic in pattern recognition. It has receivedintensive attention because of its widespread use in intelligent systems, such as wearablecomputing, mobile and internet entertainment, and video surveillance. These applicationsshare a common characteristic, namely that the sample embedded in a high-dimensional spaceand available for model training is of small size; this is known as the ‘small sample size’(SSS)problem. Over the past few decades, many algorithms have been proposed to reduce the SSSeffect and learn robust models. This thesis aims to further improve the efficiency and stabilityof SSL in practice, specifically by exploiting SSL for data embedded in high-dimensionalspaces. We consider the following aspects of this problem and propose the followingsolutions:First, we propose rank-preserving discriminant analysis (RPDA) to exploit rank orderinformation and improve discriminant learning. In particular, RPDA encodes local rankinformation of within-class samples, and discriminative information of between-class samples,under the ‘Patch Alignment Framework’. However, like other supervised manifold dimensionreduction algorithms, RPDA has several hyper-parameters, the optimal settings for which arenot trivial to choose. We therefore propose a new dimension reduction algorithm to avoid thisproblem, termed ensemble manifold rank preserving (EMRP). EMRP finds the optimal linearcombination of the alignment matrices to approximate the intrinsic manifold in the data. Weapply these two schemes to acceleration-based human activity recognition, and achieve arobust and effective low-dimensional representation.Second, we propose rank-preserving sparse learning (RPSL), which preserves the rankorder information and obtains a sparse projection matrix, and in doing so reduces theconcentration of the measured phenomena and obtains parsimony in computation. In addition,we consider minimization of classification error to facilitate classification. By utilizing aseries of equivalent transformations, we can transform the objective function of RPSL into alasso-penalized least squares problem. In addition, in our Kinect-based scene classificationstudies, we apply locality-constrained linear coding (LLC) to local SIFT features to representRGB-D samples, and classify scenes through the cooperation between RPSL and a simple classification method. Compared to other classical dimension reduction algorithms, RPSLresults in an interpretable model and saves computational costs in the testing stage.Third, we propose a novel semi-supervised classifier, termed the Hessian-regularizedsupport vector machine (HesSVM). We carefully explain the rationale for using Hessianregularization to encode the local geometry of the compact support of the marginaldistribution, and prove that using HesSVM in the reproducing kernel Hilbert space isequivalent to conducting HesSVM in the space spanned by the principal components of thekernel principal component analysis. In addition, we present a scheme for image annotation inthe cloud, in which mobile images compressed by Hamming-compressed sensing aretransmitted to the cloud, and semantic annotation is conducted in the cloud using a novelHesSVM. We conduct experiments on the PASCAL VOC’07dataset and demonstrate theeffectiveness of HesSVM for large-scale image annotation.Finally, we investigate weakly-supervised metric learning. We noticed that KISS metriclearning estimates the inverse of a covariance matrix to be unstable, and the resultingperformance can therefore be poor. Thus, we present regularized smoothing KISS metriclearning (RS-KISS), which seamlessly integrates smoothing and regularization techniques torobustly estimate covariance matrices. RS-KISS is superior to KISS because it can effectivelyenlarge underestimated small eigenvalues, and reduce overestimated large eigenvalues, in theestimated covariance matrix. In addition, the covariance matrices of KISS are estimated bymaximum likelihood (ML) estimation. It is known that with an increasing number of trainingsamples, discriminative learning based on the minimum classification error (MCE) is morereliable than classical ML estimation. Thus, a new scheme is presented, termed the minimumclassification error KISS (MCE-KISS). These two algorithms are used in thorough validatoryexperiments on the VIPeR and ETHZ datasets, and the results show that MCE-KISS is muchmore accurate and RS-KISS is computationally much more efficient. Therefore, onealgorithm needs to be chosen according to the practical situation.