Multimodal Data Representation And Applications On Content Curation Social Networks

Increasing abundance of modalities on social networks provides opportunities and challenges for multimodal data mining.Content curation social networks(CCSNs),which are booming ones,attract a large number of users to share interests with items(i.e.pins)on it.Owing to its two characteristics which are content-centered and interest-driven,utilizing multimodal data to model users and analyze user interests on it is more necessary and feasible than on other social networks.This thesis studies multimodal data representation and its applications on CCSNs.Details are organized as follows:First,an algorithm for recommending followees by combining multimodal similarity and category consistency is proposed.User behaviors and contents on the repin paths are used for pin representation,and multimodal joint similarity between pins is measured.Then homophily between users is measured by combining multimodal joint similarity and category consistency between common pins of users,and finally used for recommending followees.Experiments show that the proposed algorithm performs better than the algorithm based on the number of common pins.Second,a framework of multimodal representation based on deep learning is proposed.Dataset for learning image representations are annotated automatically with repin trees.Image representations and text representations of pins obtained by deep learning are fused into multimodal joint representations with a deep model.Experiments show that multimodal joint representations are more effective than either image or text representations on predicting interest distributions of pins.Third,three board and user models based on multimodal representations are proposed.The interest distribution model,which reveals personality on CCSNs,is the mean of interest distributions of pins.Other two models transform a set of pin vectors into a constant dimension vector to represent a board or a user.Fourth,some recommendation algorithms based on multimodal representations is proposed.They respectively recommend pins,board thumbnails,board categories,boards and users.Experiments validated their effectiveness,and evaluated the multimodal representations and three proposed models at the same time.