Visual Learning Of Pairwise Similarity And Relative Order Relationships

Many computer vision problems can be viewed as image pairwise relationship learning tasks.They learn a model to predict whether a given image pair belongs to a particular pairwise relationship.Among the existing image pairwise relationships,the similarity relationship and relative order relationship are the two most common pairwise relationships in the computer vision tasks.The similarity relationship learning method,which is also named as similarity learning method,aim to learn a proper similarity measure,with which the similarity between images can be more effectively evaluated for classification.Different from similarity,the relative order is a kind of antisymmetric relationship.The goal of relative order relationship learning is to learn a prediction model to predict the relative order relationship between two images.The similarity learning can be divided by two categories,i.e.Mahalanobis distance metric learning and deep similarity learning.The Mahalanobis distance metric learning methods measure the pairwise similarity by the Mahalanobis distance.Most of the Mahalanobis distance metric learning methods are optimized by the traditional gradient descent based method,heuristic algorithm or one-pass training.However,these algorithms are not able to take both of recognition rate and training efficiency into account.In recent years,a series of deep similarity learning methods are proposed to break through the bottleneck of hand-crafted feature.The existing deep similarity learning methods usually formulate the similarity as the Euclidean distance of the image deep feature.It is efficient in recognition,but it is not enough to represent the connection between images since it is only relevant with the difference of the deep features.The deep Siamese network is one of the important models in image pairwise relationship learning.However,the existing deep Siamese network are mainly applied into similarity learning.It’s thus important to extend the deep Siamese network to the relative order relationship learning problem.Besides,the existing relative order relationship learning methods are mainly applied into the ranking problem,but the relative order relationship also exists in the regression problem.How to apply the relative order relationship method to the regression problem is also a crucial issue.As in some applications,there are multiple relative order relationship to be learned,it’s also important to develop the multiple relative order relationship learning method to jointly learn the relative order relationships of multiple elements.In this thesis,we aim to develop the distance metric learning,deep similarity learning,single relative order relationship learning and multiple relative order relationship learning models for image pairs.The main work of this thesis includes(1)Distance metric learning based image pairwise similarity learning model.We formulate metric learning as a kernel classification problem with the positive semidefinite constraint,and solve it by iterated training of SVMs.The new formulation is easy to implement and efficient in training with the off-the-shelf SVM solvers.Two novel metric learning models,namely Positive-semidefinite Constrained Metric Learning(PCML)and Nonnegative-coefficient Constrained Metric Learning(NCML),are developed.Both PCML and NCML can guarantee the global optimality of their solutions.Experiments are conducted on UCI dataset classification,handwritten digit classification,face verification and person re-identification to evaluate our methods.Compared with the state-of-the-art approaches,our methods can achieve comparable classification accuracy and are efficient in training.(2)Combination of image representations based image pairwise deep similarity learning.We propose a novel similarity measure to introduce the pairwise image representation to better represent the connection between images.We also fuse the pairwise image representation and single image representation to combine their advantages.A convolutional neural network(CNN)based similarity learning approach is proposed to jointly learn the SIR and PIR to optimize the proposed similarity measure.Both SIR and PIR can be jointly learned for pursuing better matching accuracy with moderate computational cost.Furthermore,the similarities learned with pairwise comparison and triplet comparison objectives can be combined to improve the matching performance.Experiments on the CUHK03,CUHK01 and VIPeR datasets show that the proposed method can achieve favorable accuracy with modest training time.(3)Learning and predicting image pairwise relative order relationship based on deep Siamese convolutional network.We study to extend the deep siamese network from similarity learning to relative order relationship learning.We formulate the second-order image representation and the relative order relationship prediction function.Then we propose an extended deep siamese CNN based method with relative order loss,mean square error(MSE)loss and softmax loss to learn the relative order relationship.Furthermore,we find that the proposed method can also be applied to the regression task,e.g.age estimation,although it is not aimed at predicting pairwise relationship.We conduct the experiments on relative attribute ranking and age estimation tasks.The results show that the proposed method achieves the state-of-the-art performance,and outperforms the competing methods.(4)Joint learning and predicting image pairwise multiple relative order relationships based on deep Siamese convolutional network.We study the multiple relative order relationship learning problem for the camera pose estimation and relative attributes task.We consider the this task as an Multi-Task Learning(MTL)problem,in which the learning of each prediction component is regarded as a learning task,and we propose a multiple relative order relationships learning method based on deep siamese networks.In our proposed method,we use the second-order representation of images to learn the relative order relationship,and adopt the relative order loss and mean square error(MSE)loss to make the predicted values and their relative order to be consistent with the ground-truth.To jointly learn multiple relative order relationships of prediction values,we propose a deep siamese network which consists of two shared branches.Each branch consists of the spatial sub-network and regression sub-network,which learn the spatial feature and the regressors,respectively.The spatial sub-network is shared across all the learning tasks,and it can capture the generality between different prediction values.As the regressors of the prediction values are different,the regression sub-network of different prediction values are separated.So it can capture the specificity of each prediction value.The experimental results show that our proposed method can achieve satisfactory performance in camera pose estimation and relative attributes.