Research And Application On Supervised Similarity Metric Learning Approaches

The concept of big data is extraordinarily hot in recent years.Machine learning,as one of the most important technologies for big data analytics,has attracted an increasing interest of researchers in the different fields,and also has been widely applied in the industrial community.Similarity metric is a key issue in many machine learning algorithms,such as nearest neighbor classifier,support vector machine and Kmeans cluster.Although many off-the-shelf similarity functions such as Euclidean distance can be used,they may not be appropriate to capture the idiosyncrasies of the data of interest.A desired similarity function is usually designed specifically for the task at hand.But handcrafting such good similarity metrics is generally very difficult.This has led to the emergence of similarity metric learning,which aims at automatically learning a similarity function from data to improve the performance of the learning system.There has been considerable research on metric learning over the past few years.A typical method is learning a similarity function parameterized by a matrix with some constraint information of similarity between samples to capture the potential similarity.The goal of metric learning is not just to find a good similarity metric,more importantly is to improve the performance of the subsequent learning task.Therefore,our focus is to study how metric learning can be used to improve the performance of the specific learning task.There are still many valuable and challenging questions on the learning of similarity metric.For example,the existing methods are generally a category of two-stage method,which means that the processes of learning the similarity function and training the resultant learner are separate.More theoretical understanding is needed to guarantee that the learned metric can lead to a good generalization of the resultant learner,such as nearest neighbor classifier which is widely used.To achieve better generalization performance,large amount of training examples are usually used.Since new data always comes continually under the background of big data,effective learning algorithm is needed to update the similarity metric in the incremental situation.In addition,two-stage methods can not always improve the performance of some global learning models such as support vector machine,so one-stage similarity metric learning has attracted more attention recently.Thus,how to combine learning of the target function with the best similarity function into a single,and ideally optimization problem is also an interesting question.The main contributions of this dissertation are summarized as follows:1.Margin distribution explanation on metric learning for nearest neighbor classification: the learned metrics of many two-stage metric learning methods are in turn heavily used for the Nearest Neighbor classification(NN).Although they always achieve better classification accuracy in the experiments,until now no theoretical link has been established between the learned metrics and their performance in nearest neighbor classification.We analyze the effectiveness of metric learning from the perspective of the margin distribution and derive the generalization error upper bound for NN with respect to the Mahalanobis metric.Experiments show that large margin distribution can be obtained by these algorithms,and the margin distribution criterion can be used to design new metric learning algorithms.2.Incremental metric learning with with support vector approach: In practical applications,data always comes continually and this is the so called incremental situation,in which the metric need to be updated incrementally.However,the existing methods are inflexible to handle this problem.We propose an effective incremental metric learning algorithm by extending the previous work on classification scenario of the incremental SVMs.The algorithm can be guaranteed to learn a metric equivalent to the one learned from scratch.We provide not only a theoretical analysis for both the feasibility and the convergence of the algorithm but also an empirical verification on its effectiveness.3.Kernel embedding metric learning for regression: We propose a one-stage metric learning method with kernel embedding for the regression task.Our method combines the learning of the metric with the training of support vector regression by embedding the metric matrix into the kernel function.Considering that redundancy features or noise may exist in real data,our algorithm enforces the sparsity on the metric matrix to removing the potential redundancy or noise.We also develop a bagging-like effective ensemble metric learning to improve the generalization performance of our algorithm.It is verified on many benchmark datasets,and also applied to the problem of airport noise prediction.4.Similar feature embedding similarity learning: We propose a novel one-stage similarity learning method from the perspective of learning in the similarity space which uses the similarities between data as features.Considering that selecting effective representative samples to construct discriminative similarity features is as important as learning a good similarity function,we combines the similarity learning with the training of lasso to select the effective representative samples.The learned similarity metric is more flexible since the similarity matrix does not need to be positive semi-definite.We provide the theoretical analysis for the generalization performance of the algorithm,and also an empirical verification on its effectiveness.