Research On Classifier Design Incorporated With Prior Knowledge

Classifier design is a critical step in pattern recognition system. Its objective is to learn aclassification rule from the given observed or training data, and use such rule to predict unseen datawith good generalization performance. However, due to the limited number of training data, and theaccompanied noise in real applications, a classifier well performed on the training data may notclassify the unseen or testing data correctly. To improve the generalization performance, one shoulduse prior knowledge related to the current classification problem as much as possible. Moreover, fromthe well-known "No Free Lunch Theorem", there is no classification method with constant superiority,one should explore prior knowledge of specific classification task as much as possible to obtain the"optimal" performance. As a result, for any classifier, its generalization performance=data+priorknowledge! In this paper, we find that the discriminative function based Support Vector Machine(SVM) and its variations, which are among the most commonly-used and powerful classificationmethods, do not take full advantage of prior knowledge in some classification problems. Thereby, weincorporate the prior knowledge into learning to further boost their generalization performance. Thetypes of prior knowledge incorporated are the discriminative knowledge of feature and the clusterstructure knowledge of data, and the incorporation strategies are regularization and problemformulation. The main contribution of this paper are summarized below,1) Suggest to consider the feature discrimination information in classifier design for the first time,and develop feature discrimination incorporated SVM (FDSVM) through considering suchinformation in SVM. SVM penalizes all feature weights with an equal degree, while FDSVMpenalizes each feature weight by an amount decreasing with the corresponding featurediscrimination measure, consequently features with better discrimination can be attached greaterimportance. Experiments demonstrate that FDSVM often achieves better generalizationperformance than SVM while retains comparable efficiency.2) Develop a structure-embedded AUC-SVM (SAUC-SVM) through incorporating the globalcluster structure information in the sample pair set into AUC-SVM. AUC-SVM emphasizes moreon the local discriminative information related to the support vector sample pairs, while neglectsthe global structure information of data distribution. Moreover, sampling method adopted toreduce the high complexity of AUC-SVM would lead to a further loss of global distributioninformation. SAUC-SVM combines the local discriminative information and global cluster structure information in a unified formulation so as to compensate such information loss andfurther boost the generalization performance. Comparative experiments demonstrate thatSAUC-SVM often achieves better AUC performance than AUC-SVM with comparableefficiency.3) Point out that in multi-class classifier design based on ECOC, at least one class in each binarysub-problem is actually a "meta-class" consisting of multiple original classes, but treated as asingle class, consequently resulting in the oversight of prior knowledge in individual originalclasses. Thus we incorporate such information into classifier design and develop a modifiedmulti-class classification method based on ECOC. Comparative experiments demonstrate that themodified method can indeed obtain performance improvement, verifying the effectiveness ofconsidering such information in ECOC.4) Develop a modified cluster assumption, and a new semi-supervised classification methodSSCCM based on such assumption. Through modifying the cluster assumption from "similarinstances should share similar label output" to "similar instances should share similar labelmembership", SSCCM allows each instance to belong to multiple classes with different classlabel memberships. SSCCM returns the decision function as well as the label membershipfunction. The two predictions are usually consistent, thus verify each other, otherwise, theirinconsistency could be utilized to enhance the reliability of semi-supervised classification.Comparative experiments demonstrate that compared with semi-supervised classificationmethods based on the cluster assumption, SSCCM exhibits competitive performance, verifyingthe effectiveness of the modified cluster assumption.5) Develop a soft large margin clustering method SLMC through further applying the modifiedcluster assumption in maximum margin clustering (MMC). SLMC maximizes the marginbetween clusters, and simultaneously allows each instance to belong to multiple clusters with thecorresponding soft cluster memberships, thus combines the advantages of maximum marginprinciple and fuzzy clustering idea. Comparative experiments demonstrate that SLMC performsbetter than both MMC and the typical soft clustering method FCM.