Research On Regularization Techniques In Classifier Design

Regularization techniques were originally proposed in the 1960s in mathematics to solve the ill-posed problems. In the past decades, regularization techniques have been widely applied in various research fields of pattern recognition with the rise of machine learning. By incorporating the regularization term embedded the right amount of prior information, regularization techniques have been shown to be powerful in making the solution stable. And many famous related algorithms have been developed, such as Regularization Networks (RNs), Regularized Least-squares Classification (RLSC), Support Vector Machines (SVMs), and Manifold Regularization (MR). This thesis focuses on one of the most important phases in pattern recognition system–– classifier design, and systematically researches the regularization techniques in the classifier design from the three aspects, that is, the generalization performance of the regularized classifier, the construction of the regularization term and the integration of prior information with the classifier. The major contributions of this thesis are that:1. A novel locality regularized generalization error bound is proposed, which is based on the square error criterion classifier. The new bound limits the generalization error in the local neighborhood of each sample, and also introduces the structural information in the sample space. Consequently, the bound can overcome the shortcomings of the current bounds that can only be applied in linear classifiers, such as VC-dimension. The introduction of the tunable regularization parameter further increases the flexibility of the bound. Moreover, a new classifier design method, termed as Locality Regularization (LR), is presented from the bound. LR has two major characteristics: (1) LR naturally deduces the regularization term from the defined expected risk functional; (2) Through combining with spectral graph theory and manifold learning, LR constructs the regularization term within the locally alterable sample neighborhoods and further improves the generalization capability. Experimental results show that LR yields better classification performance than the traditional regularization methods, especially in the datasets that the training and testing samples distribute unevenly.2. A feature selection algorithm using the locality regularized generalization error bound is developed, which further extends the application fields of LR. As a hybrid filter-wrapper method, it uses the locality regularized generalization error bound as the evaluation function as well as LR algorithm as the classifier. As a result, it can not only keep high computational efficiency, but also guarantee the good generalization performance of the following classifier.3. A novel approach to construct the data-dependent regularization term called Discriminative Regularization Term (Rdisreg) is presented, which can solve the problem of the traditional regularization methods that they only pay attention to the data- independent smoothness of the classifier. Rdisreg mainly emphasizes on: (1)Through adopting different ways to define the intra-class compactness and inter-class separability in the output space, Rdisreg introduces the discriminative and structural information into the regularization term, which is vital for classification; (2) Rdisreg only has one adjustable regularization parameter and thus effectively avoids the potential"curse of dimensionality"in the mutli-class optimization; (3) Rdisreg can derive a large family of new regularization learning algorithms through combining with different loss functions and regularization terms, which provides a new way to design regularized classifiers. Two corresponding discriminative regularization (DR) methods based on the least-squares loss function are proposed, which respectively integrate the globally and locally structural information into the Rdisreg. By embedding equality type constraints in the formulation, the solutions of DR can follow from solving a set of linear equations and the framework naturally contains multi-class problems. Experimental results demonstrate the superior generalization performance and stability of DR.4. This thesis reveals the relationship between SVM and its corresponding improved algorithms from the structure granularity viewpoint. And a structurally regularized large margin classifier framework is constructed. Furthermore, a new large margin algorithm -- Structurally Regularized Support Vector Machine (SRSVM) is proposed. Through embedding the cluster structural information into the SVM objective as a new regularization term, SRSVM can: (1) hold the traditional SVM optimization framework, decrease the computational complexity, simplify the kernelization process, converge to global optimum, and keep the sparsity of the solutions; (2) achieve theoretically and empirically better generalization than SVM.5. Following the well-known"No Free Lunch Theorem", the researches on regularization techniques in this thesis all involve integrating as much prior knowledge as possible in the regularized classifier, including discriminative information and structural information. Consequently, how to mine the structural information hidden in the samples effectively is another research direction in the thesis. An alternative robust local embedding (ARLE) is presented. ARLE focuses on the robust reconstruction of the samples, and considers the global and local structure of the sample manifold simultaneously. As a result, ARLE has more compact embedding than the traditional manifold algorithms and suppresses an unfavorable influence of the outliers on the embedding process automatically. The preliminary experiments in the classifier design show that ARLE can further improve the performance of regularized classifiers.