Research On Key Technologies Of Social Relationship Extraction Of Video Characters

Character social relationship extraction aims to identify the video character entities,analyze whether there is a social relation between characters and what kind of social relation exists,and present it in a structured form.As an impor-tant task of information extraction,it can help the tasks of video understanding,character tracking,role discovery,etc.Meanwhile,it can provide great social and commercial value for knowledge discovery,public security monitoring,ad-vertising,etc.However,due to the complexity and abstraction of the video,and the number of videos is huge,it faces the following four challenges:1)How to extract complete and accurate social network from complex videos;2)How to use multi-view features to solve the gap between underlying pixels and high-level social relationship space;3)How to mine the temporal features of interac-tions between characters to improve the accuracy of social relation recognition;4)The efficient calculation of the social relationship extraction from massive videos.Therefore,in order to deal with the above challenges,the thesis proposes a series of novel models and methods from two aspects:construction of the social relationship network and character social relation recognition.Then the parallel algorithms for social relation extraction are designed and implemented.The main contributions of this thesis are as follows:(1)Character relationship network construction method based on story segmentation.First,characters are labeled by the face detection and recognition methods.Second,a story segmentation algorithm based on video hierarchical features is proposed.The algorithm divides the video into stories according to the coherence of video content features.Then,the Gaussian weighting is used to calculate the relationship weight of the person based on the story seg-mentation result.In this way,the accuracy can be improved.Furthermore,the character relationship networks extracted from the video and subtitle text are fused.Finally,comprehensive evaluations are conducted on three movies and one TV drama,and the experimental results demonstrate that the proposed method outperforms the state-of-the-art methods.(2)Social relationship recognition based on multi-view feature fusion.First,in order to solve the problem of lacking video dataset for social relation analysis,this thesis builds a new video dataset.Second,this thesis proposes two methods based on multi-view features fusion to overcome the gap between pixels and high-level social semantics,which cannot be well solved by single view.The first model is a multi-stream fusion network,which first extracts the high-level semantic features of the RGB,optical image and audio.employs the logistic regression to perform post-fusion operation to classify the social rela-tions.The second model is a multi-view fusion method based on the attention mechanism.The attention mechanism is used to assign different weights to different features,and a tensor fusion layer is designed to fuse the multi-view.(3)Social relationship recognition based on spatio-temporal fine-grained features.In this thesis,an attentive sequences recurrent network is proposed for social relation recognition from video.First,a multi-feature fusion module based on attention mechanism is employed.It combines different features at the same time from a fine-grained perspective to establish the best temporal feature description.Second,an attention sequence network based on global and local sequence features is proposed.It can automatically detects the shots boundary of the video and obtains the global feature of the video through the statistics of the shots features.The attention mechanism is introduced to automatically learn global feature.Through this manner,the video frames or clips,which can better reflect the traits of the social relationship of the character are given higher weights.Therefore,the accuracy of the character relationship recognition can be improved.(4)Design and implementation of character relationship extraction algo-riths based on parallel platform.First,a video character social relationship extraction framework based on parallel computing platform is designed.The framework supports distributed storage and management of long video and deep learning algorithms based on GPU training.In this way,the efficiency of massive video processing can be improved.Second,we design parallel algo-rithms related to video character social relationship extraction,including video keyframe extraction,feature extraction,face data analysis,character relation-ship extraction,etc.These algoriths not only guarantee the accuracy,but also reduce the time cost.Finally,the effectiveness of the framework and algorithms is verified.In general,this thesis studies the key technologies of video character social relationship extraction.Our methods and models achieve better performances than the state-of-the-art methods on the video social relationship dataset.