Large Margin Nearest Neighbor Based Distance Metric Learning

In the environment of globalization,information revolution and Internet development,massive complex structured data are continuously collected and stored.As an essential part of technological progress,intelligent data analysis has become particularly important.Correspondingly,machine learning has attracted more and more attention due to its key role in data processing.Metric learning is a fundamental problem in machine learning.Many machine learning methods,such as K-means clustering,nearest neighbor classification,support vector machine,largely depend on whether the metric can reasonably reflect the important relationship between data.Although the manually selected distance functions such as Euclidean distance have advantages of simplicity and generality,it is difficult to adapt to most problems in pattern classification.Metric learning,aiming to automatically learn a task-specific distance function to effectively calculate the similarity between the input data,has become a hot research topic in recent years.For the challenging problems of metric learning,this thesis makes a deep research on large margin nearest neighbor based metric learning model,in which the philosophy of regularization prior design,characterization of metric matrix,construction of Mahalanobis distance,and model optimization are adopted.The major contributions are the following:We propose a novel large margin nearest neighbor based metric learning model via learning the compositional structure of metric matrix M.We consider M having an inexact low-rank structure and use the sparse and low-rank compositional model to capture it.Specifically,instead of recovering the decomposition by using l₁-norm regularization over sparse part S,we control the pixel values of S less than a threshold.This will make the model more stable since the loss function treats the pixel values equally if they are bigger than the threshold.Different from trace norm which minimizes sum of all singular values over low-rank part L,we penalize the singular values of L less than another predefined threshold.As a result,the nonrelevant information associated to smallest singular values can be filtered out,such that the metric learning model is more robust.Meanwhile,we don't need to know the optimal sparsity degree and rank with regard to the metric matrix M and only input an approximate sparsity degree and rank value,thus our model is more applicable in practice.We introduce the maximum correntropy criterion into the metric learning model to deal with real-world malicious occlusions or corruptions.Unlike conventional methods,we consider the metric learning model as a reconstruction problem,and enforce the intra-class recon-struction residual of each sample to be smaller than the inter-class reconstruction residual by a large margin.Since the distance matrix M is positive semide-finite,it can be factorized as M=L L.Then the reconstruction residual can be viewed as implementing a linear transformation with projection L.To effectively cope with the noise caused by occlusions or corruptions,we utilize correntropy induced metric to characterize this projected reconstruction residual.Therefore,our model not only can handle the occlusion problem in real data but also absorbs the advantages of conventional metric learning methods,i.e.,the effectiveness in image alignment.Traditional metric learning methods aim to learn a single Mahalanobis distance matrix M,which,however,is not discriminative enough to characterize the complex and heterogeneous data.Besides,if the descriptors of the data are not strictly aligned,Mahalanobis distance would fail to exploit the relations among them.To tackle these problems,we propose a multi-level metric learning method using a smoothed Wasserstein distance to characterize the errors between any two samples,where the ground distance is considered as a Mahalanobis distance.Since smoothed Wasserstein distance provides not only a distance value but also a flow-network indicating how the probability mass is optimally transported between the bins,it is very effective in comparing two samples whether they are aligned or not.In addition,to make full use of the global and local structures that exist in data features,we further model the commonalities between various classification through a shared distance matrix and the classification-specific idiosyncrasies with additional auxiliary distance matrices.Most of the existing metric learning methods input the features extracted directly from the original data in the preprocess phase.What's worse,these features usually take no consid-eration of the local geometrical structure of the data and the noise existed in the data,thus they may not be optimal for the subsequent metric learning task.We integrate both feature extraction and metric learning into one joint optimization framework and propose a new bilevel distance metric learning model.The lower level focus on detecting the underlying data structure,while the upper level directly forces the data samples from the same class to be close to each other and pushes those samples from different classes far away.Note that the input data samples of the upper level are represented by the sparse coefficients learnt from the lower level model.And benefits from the feature extraction operation of the lower level model,the new features become more robust to noise with the sparsity norm and more discriminative with the Laplacian graph term.In summary,on the basis of the fundamental theory of signal processing and pattern recognition,this thesis takes statistical learning as the main investigative means and proposes four novel metric learning methods.The proposed methods can effectively overcome the limitations of the existing methods and achieve better results on face verification and pattern classification,providing a new approach to metric learning task.