Research On Affective Computing And Applications Of Image Emotion Perceptions

With the rapid development of computer science, multimedia technology and social networks, the volume of multimedia content is growing explosively, resulting in great de-mand of processing and understanding the visual content of images and videos. Humans are able to perceive and understand images and videos only at high level semantics, in-cluding cognitive level and affective level, rather than at low level visual features. Most previous works on image content analysis focus on understanding the cognitive aspects of images, namely describing the actual content, such as object detection and recognition. However, with the increasing use of digital photography technology by the public and the high requirement for image emotion representation, the analysis of image content at the highest semantic level, i.e. the affective level, is becoming increasingly urgent.The analysis of image content at the affective level, abbreviated as image emotion computing, aims to understand the emotional reactions of users after viewing give images. The development of image emotion computing is limited by two main challenges. One is the affective gap, which can be defined as "the lack of coincidence between the measur-able signal properties, commonly referred to as features, and the expected affective state in which the user is brought by perceiving the signal". The other is the subjectivity of im-age emotion perceptions and evaluations, which can be considered as the fact that "due to the influence of cultural backgrounds, educational status and social context, the emotions that are evoked in different viewers by an image are highly subjective and different". This dissertation works on the above challenges of image emotion computing, which expects to extract emotion features that are more discriminative and interpretable based on related art theory; predicts user-centric personalized emotion perceptions using social media data and explores the factors that can influence emotion perceptions; models the probability distribution of image emotions, which predicts the emotion distribution of an image when viewed by large populations; investigates the applications of image emotions in computer vision and multimedia technology. Specifically, the main contents and contributions of this dissertation can be summarized as the following four aspects:Firstly, according to related art theory, this dissertation proposes mid-level principles-of-art based emotion features to predict image-centric dominant emotions. Art theory is composed of elements-of-art and principles-of-art. Elements-of-art are the building blocks or ingredients used by artists to create an artwork, including color, texture, etc, while principles-of-art are the rules and tools of arranging and orchestrating the elements-of-art in an artwork, including balance, emphasis, etc. Most existing works target low lev-el visual features based on the elements-of-art for image emotion recognition, which are not invariant to their different arrangements and their link to emotions is weak. Therefore, elements must be carefully arranged and orchestrated by principles-of-art into meaning-ful regions and images to describe specific semantics and emotions. This dissertation proposes to study, formulate, and implement the principles-of-art systematically, based on the related art theory and computer vision research. The quantized principles are combined together to construct image emotion features for affective image classification and regression. The experiments conducted on the IAPS, Abstract and ArtPhoto datasets demonstrate the effectiveness of the principles-of-art based emotion features.Secondly, this dissertation proposes to predict user-centric personalized emotion per-ceptions of social images, which is the first work to evaluate the subjectivity of emotion perceptions. Existing datasets on image emotions are mainly image-centric, aiming to predict the dominant emotions. Besides, the number of images in these datasets is small, which cannot be used for personalized emotion analysis. This dissertation sets up a large-scale image emotion dataset from Flickr, named Image-Emotion-Social-Net (IESN), with over 1 million images and about 8,000 users. The personalized emotion perceptions in social networks can be influenced by different types of factors, including visual content, social context, temporal evolution, and geographic location. Rolling multi-task hyper-graph learning is presented to consistently combine these factors and a learning algorithm is designed for automatic optimization. The experimental results on the IESN dataset show that jointly considering different types of factors can significantly improve the pre-diction performance of personalized emotion perceptions.Thirdly, from a new perspective for image-centric emotion modelling, this disser-tation proposes to predict the probability distribution of image emotions. The statistical analysis from the Abstract and IESN datasets show that though personalized, the image emotion perceptions follow certain distributions. Based on this observation, the emotion distribution prediction is formulized as a shared sparse leaning problem, which is opti-mized by iteratively reweighted least squares. Corresponding to categorical and dimen-sional emotion representations, both the discrete and continuous probability distribution are modelled and predicted. Besides, this dissertation introduces several baseline algo- rithms. Experimental results show that shared sparse learning outperforms the baselines for emotion distribution prediction.Finally, this dissertation implements several applications of image emotions in com-puter vision and multimedia technology areas. One is affective image retrieval via multi-graph learning. Different from traditional content based image retrieval, this dissertation retrieves images from an affective perspective using multi-graph learning, which is ex-tended in 3D object retrieval. One is video classification and recommendation based on affective analysis of viewers. This dissertation presents a novel method to classify and recommend videos based on affective analysis, mainly on facial expression recognition of viewers. One is emotion based image musicalization. Appropriate music with approxi-mate emotions to the given images is selected to musicalize the images, which can make the images vivid and bring people into their intrinsic world.Through the above studies, this dissertation deeply explores the different aspects of image emotions, providing feasible and effective solutions towards the key technical issues of image emotion computing. The experimental results show that:incorporating the research of related subjects such as art theory can help to extract more discriminative and interpretable emotion features and thus improve the recognition performance of image emotions; the image emotion perceptions in social networks are personalized, which are influenced by various factors, such as temporal evolution and social context, and jointly considering these factors can significantly improve the emotion prediction performance; as a trade-off between personalized emotions and dominant emotion, modelling image emotions from the point of view of probability distribution is more accordant with the actual situation and has more practical implications.