Social Relationship Understanding In Visual Content

Social relationships are abstract and non-intuitive psychological relationships,whose essence is to reflect the distance between human individuals.Usually,the exact social relationship can be inferred by some cues,such as human condition,behavior,environment,and so on.Social relationship understanding in visual content aims to judge whether human pairs has kinship or not and recognize the exact social relationship,then present them by the graph.Now,research on social relationship understanding is still in the primary stage.It faces the following four challenges: 1)How to verifying the kinship between face image pairs accurately;2)How to generate the family tree for a given family photo automatically;3)How to promote social relationship recognition accuracy by abstracting multi-source attribute features;4)How to recognize social relationships by using human attributes more efficiently.To deal with the above problems,this thesis proposes a series of social relationship understanding methods by deep learning,metric learning,feature selection,and knowledge distillation algorithms.The main contributions of this thesis are summarised as follows:(1)A novel deep metric learning model is proposed for kinship verification.Most existing kinship verification methods exploit shallow models to extract linear features but ignore nonlinear features.The nonlinear features obtained by deep learning algorithms are more abstract and semantic,and can better describe the rich internal information.This model is proposed by integrating deep learning architecture into a metric learning algorithm,which employs a deep learning model to select nonlinear features,and aims to find the appropriate project space to ensure the margin of sample pairs without kinship as large as possible and the margin of sample pairs with kinship as small as possible,simultaneously.Compared with the shallow models,this deep metric learning model promotes the accuracy of kinship verification.It indicates that this model effectively utilizes nonlinear features to verify kinship.(2)A novel deep kinship verifying and recognition framework is proposed for multiperson kinship matching and recognition.Compared with most existing kinship understanding methods that mainly work on verifying kinship in pairwise face images,we target at recognizing the exact kinship in nuclear family photos consisting of multiple persons.Firstly,we design a deep kinship matching model to verify face pairs in family photos have kin or not by using the similarity of kin pairs.Secondly,we develop a deep kinship recognition model to predict the exact kinship categories,in which gender and relative age attributes are utilized.Thirdly,we propose a family tree inference model to generate the most reasonable family tree by exploiting the kinship rules.Experiments on three kinship datasets with family structure demonstrate that the proposed method achieves excellent performance for kinship verification and recognition.Especially,the proposed family tree inference model fully exploits the common kinship rules to well boost matching and recognition accuracy and ensure the output family tree is reasonable.(3)A novel feature selection framework is proposed to recognize social relationships.Social relationships belong to the psychological relationship,which can not be observed directly.To recognize exact social relationships usually depends on human attributes including age,gender,emotion,pose,and so on.These multi-source heterogeneous attributes may contain redundancies and noises,which may cause over-fitting,low efficiency in the training phase.To solve the above problems,we propose a feature selection algorithm that selects a feature subset from the multi-source heterogeneous features by learning a sparse weighting matrix.The experimental results demonstrate the proposed method well boost the accuracy of social relationship recognition based on face and body attributes by removing noises and redundancies.(4)A novel knowledge distillation model is proposed for social relationship recognition.The great cost is expended to acquire and label human attributes related to social relationships.And the existing social relationship datasets do not provide labels of relevant human attributes.Under the shortage of attribute labels,a knowledge distillation algorithm is designed for transferring the knowledge of pre-trained human attribute models to the social relationship recognition model.This model aims to preserve the knowledge of human attributes in the learned features.The proposed model achieves state-of-the-art performance for social relationship recognition task based on face and body attributes,which take advantage of human attribute knowledge efficiently.Moreover,the proposed method can carry many mainstream neural network models to boost the performance of the original model with the help of auxiliary models.It can be seen that the architecture of the proposed model is flexible and extensible.