Analyzing social relationship in visual social media

My thesis focuses on analyzing image-based social relations of people in a given photo. Compared to general face recognition problems, social relation analysis between facial images is a new research topic. There are two parts in my thesis: first part deals with pair-wise kinship verification, and second part deals with group photo analysis for social relation. Because of the importance of age estimation in group photo analysis, I propose a novel age estimation scheme to improve the estimation performance. Furthermore, I also study the scheme to minimize the influence of gender and race for age estimation.;Most current studies are based on measuring the appearance similarities between global faces. However, most significant cues for kinship verification are from local facial regions instead of the global face. My study solves this problem by finding these local appearance cues. Meanwhile, to improve the verification performance, I also build a 2D face geometry model to measure the face shape similarity between facial images. Comprehensive evaluations are performed for my fused appearance and geometry model.;In everyday life, people usually take photos together. Predicting the relationship of group of people thus becomes an essential topic for image-based social relation inference. It has several applications since family is one of the most important social units in the society, thus categorizing family photos constitutes an essential step towards image-based social analysis and content-based retrieval of consumer photos. Our model aims to capture the characteristics of group photos from different aspects. We propose an approach that combines multiple unique and complimentary cues for recognizing family photos. I first propose a geometry model to characterize people's standing pattern. To better measure the facial similarity, I apply deep neural network for measuring the appearance similarities between different individuals. I also apply the scene understanding knowledge to improve the group photo recognition performance. Experimental results demonstrate that age information plays an important role in the group photo categorization.;To improve the performance of age estimation, I propose a new aging feature extraction scheme via Convolutional Neural Network (CNN). The new aging pattern is learned through the image data instead of using manually-crafted features. My proposed aging pattern consists of the feature extracted from many hierarchical layers instead of only using the top-layer. To improve the age estimation performance and the efficiency, I apply manifold learning to project the extracted aging feature into another low-dimensional feature subspace. I also evaluate the performance using different regression and classification methods to predict the final age value.;One drawback of deep neural networks is its complicity and model size. This especially poses a challenge for deploying the learned model on mobile devices. To get a smaller model with a comparable performance, I advocate a new unsupervised aging feature via convolutional sparse coding. Experimental results demonstrate that our learning approach better extracts localized subtle aging features instead of losing them in a deep layered network like CNN, and also significantly reduces the memory consumption.;The performance of the age estimation system is usually influenced by many factors, such as expression, gender, and ethnicity. In this thesis, I present a new scheme to mitigate the influences due to race and gender in the problem of age estimation. I apply the correlation learning scheme on two different feature subspaces to learn the correlation projection matrix. Then I utilize discriminant learning on the projection subset learned by the correlation learning. As illustrated by experimental results, the proposed model improves the performance of age estimation across different age groups significantly.