Research On Emotional Semantics-oriented Image Understanding Based On Machine Learning

With the rise of visual media and the advent of the era of reading pictures,images have become an important tool for information exchange.Emotion-based image understanding aims to analyze and excavate the emotional semantics of images.Because emotion information plays a key role in human perception,reasoning,creation and other activities,the study on emotional semantic analysis for images is of great significance in today’s visual age.Machine learning is an effective means of emotional semantic analysis for images,but the semantic gap and the subjectivity of emotion cause difficulties for emotion-related image understanding based on machine learning.Moreover,existing research on this topic often neglects the difference of emotion generation mechanism between abstract images and non-abstract images.In view of these problems,this dissertation,on the one hand,conducts research on emtiom-related abstract image understanding by creating direct mappings between image features and emotional semantics.A novel feature extraction technique,unsupervised feature learning,is first studied and then applied to emotional semantic analysis of abstract images by means of transfer learning.On the other hand,this dissertation takes the non-abstract images freely shared by users in social networks as the research object,and conducts research on image sentiment prediction based on intermediate ontologies.The main innovative achievements in this dissertation include:(1)The relationship between the whitening transformation methods and the pooling algorithms in convolutional sparse autoencoder-based image classification is revealed,where the whitening transformation is always used in sparse autoencoder-based unsupervised feature learning and the pooling operation is often employed by convolutional neural network-based feature extraction.As a result,average pooling is able to provide better image classification performance when whitening processing is adopted,but max pooling provides better performance when whitening processing is unadopted.Furthermore,a convolutional sparse autoencoder-based scheme for unsupervised feature learning and image classification in YUV color space is presented.Aiming at the characteristic that the luminance component and chrominance component are independent of each other in YUV color space,a whitening algorithm treating the luminance and chrominance separately is adopted.Experiments show that the unsupervised feature learning in YUV space can achieve comparable image classification performance to that in RGB space,as long as the luminance data is properly whitened.(2)A novel image classification scheme using convolutional sparse autoencoders and domain adaptation is proposed for affective abstract painting classification in small sample size situations.First,image features are learned from a large unlabeled dataset,and then the features are extracted from the small sample size abstract painting dataset for emotional classification by means of knowledge transfer.Experiments show that the sparse autoencoder-based unsupervised feature learning techniques can be applied to image classification at the cognitive level and at the emotional level as well.Features learned from the large amount of data outside the target domain through knowledge transfer and domain adaptation can achieve better image classification performance,under the premise that the number of samples in the target domain is limited.(3)A scheme based on cross-domain convolutional sparse autoencoders is proposed for emotional textile image classification.A correlation analysis-based feature selection method for filtering the features learned by sparse autoencoders is further proposed to reduce the dimension of image features extracted by convolutional networks.Experimental results show that the sparse autoencoder-based unsupervised feature learning technique is effective in emotional textile image classification.Moreover,the appropriate feature selection can greatly reduce the time consumption of feature extraction based on convolutional networks,under the premise of slightly improving the image classification performance.(4)A VSO(Visual Sentiment Ontology)and SentiBank-based image sentiment prediction method using textual sentiment information as a supplement is proposed to make full use of the sentiment information of ontology concepts in intermediate ontology-based image sentiment analysis for social media.In addition,a regularized logistic regression model is employed to improve the performance of image sentiment prediction using the presence of sentiment ontology concepts as mid-level representations.Late fusion is adopted to combine the method using textual sentiment information with the traditional method based on intermediate ontologies.Experiments show the feasibility of using the textual sentiment information of ontology concepts,and the late fusion method even obtains comparable prediction performance to the deep learning methods.