Supervised And Semi-Supervised Multi-View Maximum Entropy Discrimination Research

In recent years, multi-view learning has drawn much attention in the field of machine learning. Usually, there are multiple views in many real-world applications. It has been proved from theory and practice that multi-view learning can improve the learning performance. Maximum entropy discrimination (MED) is a general framework for discriminative estimation based on maximum entropy and large margin principles. It possesses the advantages of flexible probability modeling and superior discriminative classification performance. Therefore, MED is a successful combination of generative learning and discriminative learning. MED spans a large majority of generative models that populate the machine learning community. Moreover, it can subsume support vector machine (SVM) as a special case by making some specific assumptions. However, up to date, it can only deal with single-view learning situation and can not effectively utilize the multi-view information for multi-view task. This thesis just right focuses on multi-view maximum entropy discrimination research and finally extends multi-view maximum entropy discrimination to semi-supervised learning scenario.In this thesis, we propose five different methods for multi-view maximum entropy discrimination. The first one is named as Multi-view maximum entropy discrimination (MVMED). It enforces the margins for two views to be identical, which means that the classification confidences form different views are deemed to be match each other. It assumes a joint probability distribution for the classifier parameters corresponding to two views and the common margin vector. Then, we design the optimization problem model for MVMED, give its solution theorem, and provide an instantiation of MVMED.The second one is named as flexible multi-view maximum entropy discrimination (FMVMED). Considering the restrict assumption of a joint probability distribution for the classifier parameters corresponding to two views and the common margin vector, it modifies the assumption as a joint probability distribution for the classifier parameter corresponding to one view and the common margin vector, and additionally assumes the posteriors for the two view margin vectors to be equal. Two joint probability distributions corresponding to two views will appear in FMVMED, making FMVMED more flexible, because there will be two Kullback-Leibler divergences in the objective functions, we can utilize a tradeoff parameter to balance the importance between the two views. We design the optimization problem model for FMVMED, make some inference and proof, and provide a detailed solving procedure which can be divided into two steps. The first step is solving the optimization problem without considering the equal margin posteriors from two views, and the second step is considering the equal posteriors. Then, we explore the relationships of FMVMED to MVMED and SVM-2K.The third one is named as multi-kernel maximum entropy discrimination for multi-view learning (MKMED). It first utilizes the linear kernel combination and then integrate the kernel combination into MED. This is a try to multi-view maximum entropy discrimination with multi-kernel learning.The fourth one is named as consensus and complementarity based maximum entropy discrimination for multi-view learning (MED-2C). It models the consensus between different views in the feature level:the two different views are transformed into a common subspace, and are made as close as possible in the new subspace. We further augment the transformed representation with the original features from each view to consider the complementarity principle. In this way, we will obtain a richer feature augmentation representation. We integrate the feature augmentation representation into MED and additionally enforce some constraints to obtain our MED-2C. Therefore, MED-2C takes good advantages of consensus and complementarity principles which will be usually respected in multi-view learning. We provide an instantiation of MED-2C and its kernel version. An alternating optimization algorithm is given.The fifth one is named as semi-supervised multi-view maximum entropy discrimination with expectation Laplacian regularization (SMVMED). It is different from the former four methods, which are all supervised learning methods. SMVMED is a semi-supervised learning method. It integrates an expectation Laplacian regularization into MVMED to utilize the geometry information of the marginal distribution embedded in unlabeled data to improve supervised learning performance.To evaluate the proposed four supervised multi-view maximum entropy discrimination methods and one semi-supervised multi-view maximum entropy discrimination method, extensive experiments on multiple real-world data sets are conducted, which demonstrate the effectiveness of the proposed methods.