Research On Deep Learning Based Multi-view Representation Learning Techniques

With the proliferation and evolution of Internet and Internet of Things,multi-view data is constantly emerging in many real-world applications.Multi-view data consists of multiple kinds of feature subsets or modals.For example,a vehicle in the automatic driving system makes decisions via the received multi-view data,i.e.,the data from multiple sensors such as camera and laser radar.By integrating consistency and complementarity involved in multiview data,we could boost the performances of many data-mining tasks.Since multi-view data usually contains more dimensions than single view data,multi-view learning tasks are more susceptible to “curse of dimensionality”.Therefore,dimensionality reduction and high level representation learning are crucial in multi-view learning.In recent years,deep learning has received significant attention in multi-view learning.However,previous works still have some problems to be solved:(1)Capturing consistence and complementarity concept in multi-view data.In multi-view data,different views contain relevance and discrepancy,we should maximize consistent information and maintain complementary information to comprehensively and accurately describe data.(2)Incomplete multi-view data.Most previous works assumed that each view has complete data.However,in real-world datasets,it is often the case that a view may contain some missing data,resulting in the incomplete multi-view clustering problem.As a result,the lack of partial views makes most multi-view clustering methods inevitably degenerate or even fail.(3)Exploiting multi-view learning paradigms to promote the traditional single view methods in many real world applications.This thesis concentrates on the above problems and proposes several novel multi-view deep representation methods.Special research topics include:1)Multi-view concept learning via deep matrix factorization.Most previous matrix factorization based multi-view representation learning methods are shallow models which neglect the complex hierarchical information.The recent proposed deep multi-view factorization models cannot explicitly capture consistence and complementarity in multi-view data.This thesis presents the Deep Multi-view Concept Learning(DMCL)method,which hierarchically factorizes the multi-view data,and tries to explicitly model consistent and complementary information and capture semantic structures at the highest abstraction level.This thesis explores two variants of the DMCL framework,DMCL-L and DMCL-N,with respectively linear/nonlinear transformations between adjacent layers.This thesis proposes two block coordinate descent based optimization methods for DMCL-L and DMCL-N.This thesis verifies the effectiveness of DMCL on 3 real-world datasets for both clustering and classification tasks.2)Incomplete multi-view clustering via generative adversarial network.Previous methods for incomplete multi-view clustering problem have at least one of the following drawbacks:(1)employing shallow models,which cannot well handle the dependence and discrepancy among different views;(2)ignoring the hidden information of the missing data;(3)dedicated to the two-view case.To eliminate all these drawbacks,this thesis presents the Adversarial Incomplete Multi-view Clustering(AIMC)framework.In particular,AIMC seeks the common latent representation of multi-view data,which is used to reconstruct the original data and infer the missing data.The element-wise reconstruction and the generative adversarial network are integrated to evaluate the reconstruction.They aim to capture overall structure and get a deeper semantic understanding respectively.Moreover,the clustering loss is designed to obtain a better clustering structure.This thesis explores two variants of AIMC,namely,AAIMC(Autoencoder based AIMC)and GAIMC(Generalized AIMC)with different strategies to obtain the multi-view common representation.Experiments conducted on6 real-world datasets show that AAIMC and GAIMC perform well and outperform baseline methods.3)Users' multi-view preferences based personalized recommendation.Recently,researchers have started to utilize the massive user-generated multi-modal contents to improve recommendation performance.However,previous methods have at least one of the following drawbacks:(1)employing shallow models,which cannot well capture high-level conceptual information;(2)failing to capture personalized user visual preference.This thesis presents a deep Users' Multi-modal Preferences based Recommendation(UMPR)method to capture the textual and visual matching of users and items for recommendation.This thesis extracts textual matching from historical reviews.This thesis constructs users' visual preference embeddings to model users' visual preference and match them with items' visual embeddings to obtain the visual matching.This thesis applies UMPR on two applications: restaurant recommendation and product recommendation.Experiments show that UMPR outperforms competitive baseline methods.4)Multi-view social media data based image retrieval.This thesis is concerned with using user-tagged images to learn proper hashing functions for image retrieval.The benefits are two-fold:(1)obtaining abundant training data for deep hashing models;(2)tagging data possesses richer semantic information which could help better characterize similarity relationships between images.However,tagging data suffers from noises,vagueness and incompleteness.Different from previous unsupervised or supervised hashing learning,this thesis proposes a novel weakly-supervised deep hashing framework which consists of two stages:weakly-supervised pre-training and supervised fine-tuning.The second stage is as usual.The framework is general and does not depend on specific deep hashing methods.Empirical results on real world datasets show that they significantly increase the performance of state-of-the-art deep hashing methods.Finally,the thesis concludes these works and discusses future works on multi-view deep learning.