Research And Application Of Social Relation Recognition In Images Based On Scene Context And Pose

Discovering social relations in images can make machines better understand human behaviors and emotions.However,automatically recognizing social relations in images is a challenging task due to the semantic gap between visual content and social relation.Existing studies usually process individual features such as faces attributes,body appearance,or objects in the scenes,to model the correlation between visual content and social relation,which cannot effectively bridge the semantic gap.Towards this end,we propose a Multi-Granularity Reasoning framework(MGR)which considers both global scenes and detailed poses of persons.The framework comprehensively captures the multi-granularity features,such as scenes,actions from persons to objects,and interactions between paired persons.By this means,our framework can close the gap between visual content and social relation to achieve precise social relation prediction.In particular,the MGR first extracts global features and local features from the whole scenes and the regions of persons and objects,respectively.Then,we innovatively propose the pose-guided Person-Object Graph and Person-Pose Graph to model the actions from persons to objects and the interactions between paired persons,respectively.Based on these graphs,the correlation between visual content and social relation is mined through graph convolutional networks.Finally,the global features and local features are integrated as a comprehensive representation for social relation classification.Extensive experiments on two public datasets show the effectiveness of the proposed framework.In addition,we design a social relation recognition system based on the MGR framework.This system includes a web service and a web application.The web service system can flexibly receive various network requests from front-end applications and provide the online social relation prediction service.Based on this service,we design and implement a prototype application of intelligent social album.This application can automatically divide personal photos into couple albums,family albums,etc.Through extensive testing,this application achieves excellent recognition results.