Research On Robust Regression And Metric Learning

In many machine learning and pattern recognition tasks,it is very important to measure the similarity between two data examples faithfully.The similarity results significantly affect the accuracy of subsequential recognition tasks.On the one hand,in real-world data,samples are usually disturbed by noise and outliers.The traditional simple measurement methods such as Euclidean distance are sensitive to noise and cannot faithfully measure the similarity between data examples.On another,for a supervised learning task,the test data may be very different from the training data,and the model generalization ability beyond the training data is not always satisfactory.To deal with the above problems,this paper makes improvements and extensions for the existing distance metrics from the perspectives of “being robust to noise” and “improving generalization performance”.Specifically,we firstly present two parameter-free robust distance metrics and apply them to robust regression tasks to achieve the purpose of being robust to noise.Then we investigate the disadvantages of existing distance metric learning models in generalization ability,and three improved learning algorithms are proposed.Our main results are summarized as follows:1.Aiming at the problem of structural noise(such as occlusion,illumination changes,etc.)in image classification,we propose a Low-rank Latent Pattern Approximation(LLPA)model,which directly solves the intermediate variable between the reference example and the test example.We employ Frobenius-norm to characterize the distance between the latent pattern matrix and the reference example,while using nuclear norm to characterize the distance between the latent pattern and the test example.After that,we obtain a distance metric that is robust to noise in the test data.We further extend the model into a regression form,which is used to measure the distance between the example and the category.An improved alternating direction multiplier method with convergence guarantee is designed to solve the regression coefficients and latent pattern matrix.Extensive experiments on multiple datasets demonstrate that our method is superior to existing reconstruction-based regression models on recognition tasks with structural noise.2.Most existing l₀-norm based distance metrics ignore the correlation of structural noise,and they usually employ approximated solutions by iteration algorithms and cannot efficently obtain an accurate solution.This paper first generalizes the l₀-norm,extending the count of non-zero elements in the norm to the count of non-zero neighborhoods,and proposes a novel ?-norm to characterize structural noise in the regression model.We utilize the kernel method to construct a kernel function that can infinitely approximate the?-norm in the kernel space,and apply it to a specific regression model to obtain an efficient closed-form solution.The recognition and reconstruction experiments on multiple image datasets demonstrate that our method is superior to most existing robust regression models in both accuracy and efficiency.3.Existing metric learning models are directly learned from the given raw training data,and they can hardly achieve good generalization performance on test data with large diversities.The Adversarial Metric Learning(AML)model is proposed in this paper,which automatically generates adversarial pairs to remedy the sampling bias and facilitate robust metric learning.Specifically,AML consists of two adversarial stages,i.e.,confusion and distinguishment.In the confusion stage,the ambiguous but critical adversarial data pairs are adaptively generated to mislead the learned metric.In the distinguishment stage,a metric is exhaustively learned to try its best to distinguish both adversarial pairs and original training pairs.So the discriminability of AML can be significantly improved.The experimental results on toy data and practical datasets clearly demonstrate the superiority of our method.4.In traditional metric learning,a single unified projection matrix can hardly characterize all data similarities accurately.To address this problem,we propose a novel method dubbed “Data-Adaptive Metric Learning”(DAML),which constructs a data-adaptive projection matrix for each data pair by selectively combining a set of learned candidate matrices.As a result,every data pair can obtain a specific projection matrix,enabling the proposed DAML to fit the training data and produce discriminative projection results flexibly.The model of DAML is formulated as an optimization problem which jointly learns candidate projection matrices and their sparse combination for every data pair.Furthermore,we extend the basic linear DAML model to the kernelized version(denoted“KDAML”)to handle the non-linear cases,and the Iterative Shrinkage-Thresholding Algorithm(ISTA)is employed to solve the optimization model.Intensive experimental results on various applications,including retrieval,classification,and verification,demonstrate the superiority of our algorithm.5.Traditional metric learning methods usually calculate the pairwise distance with fixed distance functions(e.g.,Euclidean distance)in the projected feature spaces.However,they fail to learn the underlying geometries of the sample space,and thus cannot exactly predict the intrinsic distances between data points.To address this issue,we first reveal that the traditional linear distance metric is equivalent to the cumulative arc length between the data pair's nearest points on the learned straight measurer lines.After that,by extending such straight lines to general curved forms,we propose a Curvilinear Distance Metric Learning(CDML)method,which adaptively learns the non-linear geometries of the training data.By virtue of the Weierstrass theorem,the proposed CDML is equivalently parameterized with a 3-order tensor,and the optimization algorithm is designed to learn the tensor parameter.Theoretical analysis is derived to guarantee the effectiveness and soundness of CDML.Extensive experiments on the synthetic and real-world datasets validate the superiority of our method over the state-of-the-art metric learning models.