Actor Correlation Mining In Videos Based On Spatiotemporal Context

With the development of information technology, more and more images and videos appear in our lives and the Internet. It is a challenging problem how to effectively manage and store them. Analysis and classification of the multimedia data based on their contents supply a promising approach for solving this problem. In most videos, actor correlations are essential cues to help audiences in understanding video scenario. This issue is especially for films and TV series, in which actor correlations can always include higher level semantic cues to reveal the story scenarios.Over the past few decades, many researchers have adopted actor information to analyze video content, such as actor video shots retrieval and cast listing. However, users may not pay close attention to video shots list including one actor rather than actor concurrent relationships in the huge volumes of video data.Mining actor correlations from TV series enables semantic level video understanding and facilitates users to conduct correlation based query. In this paper, we present an actor correlations mining method to build relation network based on actor concurrence parsing and visual content analysis techniques, which serves as the first attempt for effective actor association presentation and concurrence search. Our work is as follows:Firstly, a spatiotemporal context based correlation analysis is carried out to mine correlations between different people. We leverage face detection and tracking to locate actors with 2D PCA detector as pretreatment.Secondly, we propose an effective spatiotemporal descriptor to characterize human correlation contexts in a given video sequence. Our proposed is then obtained by further partition each video into more semantically meaningful clip collections using graph partition model, which can obtain more precise actor correlation graph comparing with previous works in literature. Such descriptor enables us to mine actor correlations within TV series, which facilitates semantic level video understanding, and provides users a novel actor correlation based browsing and retrieval interface.Finally, we have deployed our descriptor onto"Friends"TV series dataset (with 20 hours videos), with extensively user studies to demonstrate our performance. In concurrence graph mining, we consider video structure and hierarchical concurrence to refine actor association in our graph modeling. We present the actor correlation mining results in a graph based interface to enable efficient users'navigation and search.