Social Media Data Semantic Understanding And Associative Expression

The recent years have witnessed the explosive growth and ubiquity of mobile smart devices.The high resolution cameras,large storage,and high speed network connec-tion of mobile devices have founded the superior conditions for capturing and sharing.Users can capture and share photos or videos at almost anytime and anywhere.Up to now,social media data have increased to a huge scale.However,these data exist in a fragmented way on social media,lacking intelligent services to organize them.Neither can social media provide data according to personalized user needs,nor can users search for their required data efficiently and effectively.As a result,how to exploit and utilize the large scale social media data has become an important problem.This dissertation probes into the semantic understanding and associative expres-sion of social media data.The aim is to implement an intelligent system that can under-stand,select,and express social media data in an associative way.Due to the wide range of semantics,it's hard to collect and label data for every semantic tag.Semantic under-standing should solve the difficulty of labeling.Besides,semantic understanding should accelerate the processing speed due to the large scale of social media data.Based on the semantic understanding,this dissertation studies associative expression from photo and video aspects.Semantic understanding and associative expression compose a relatively complete framework for mining and utilizing social media data.This dissertation conducts deep research on social media data semantic understand-ing and associative expression related problems with the following achievements:1.For the labelling difficulty,proposed a weakly supervised relevance feed-back deep learning algorithm to learn from social media data directly.Traditional deep learning algorithms are sensitive to label noises.The proposed algorithm utilizes perceptual consistency to attenuate the sensitiveness,which uses the correlation in the feature space to make different samples contribute differently during training.Experi-mental results with comparison to existing algorithms show that the relevance feedback algorithm is more robust to label noises.2.For the processing speed,proposed a fast large scale high dimensional on-line feature selection algorithm to reduce the type and number of features.Based on second-order online learning,the algorithm selects features according to feature con-fidence.This dissertation proposes fast algorithms with the heap structure.Due to the monotonous increasing property of confidence,complexity of the proposed online feature selection algorithm is further reduced to be linear to the number of non-zero features.Experiments with comparison with state-of-the-art algorithms show that the proposed algorithm can significantly reduce the training time while achieving compara-ble or even higher accuracy.Second,this dissertation proposed a deep convolutional neural network model simplification algorithm based on online feature selection to accelerate feature extraction.The algorithm adds a weighting layer corresponding to each channel of the output feature map of each convolutional layer.The group sparsity problem on the three dimensional convolutional kernels is then transformed into the on-line feature selection problem on the one dimensional weighting vector.Experiments show that model parameters can be significantly reduced with little impact on accuracy.3.For associative photo expression,proposed a theme-based photo story-telling system—Monet.The system first detects events in photos and selects a rep-resentative photo subset to summarize the photo collection.After that,the system as-signs a theme to each photo according to the semantic understanding of photos.Finally,a fancy photo storytelling video is generated according to the computational filming grammars of each theme.Experimental results show that the proposed system provides better photo summarization and storytelling abiliity.4.For associative video expression,proposed an automatic mobile multi-camera recordings mashup system—MoVieUp.The system is composed of audio and video mashup.For audio mashup,the system assesses audio quality,selects high quality audio segments,and stitches them into a single audio stream.For video mashup,the system detects the switching points according to the tempo and semantics of audio.It then selects video shots by maximizing video shot quality and diversity under the constraint of motion consistency.Experimental results show that the proposed system provides better user experience for mobile multi-camera recordings.