Spatio-temporal Context Method For Complex Human Activity Analysis

In recent years,video based human activity analysis has become one of the most attractive research fields in computer vision and machine learning,because of its wide potential applications and academic values.According to the requirement of practical application in complex scenes,the task of human activity analysis not only requires to recognize category of human activity,but also requires to locate spatio-temporal boundary of human activity.Due to the complexity of human activity in real scenes,considering spatio-temporal context is benefit for improving recognition and localization's performance.Therefore,we mainly focus on how to take fully advantage of spatio-temporal contextual information for human activity analysis.In this paper,our research includes:(1)Spatio-temporal description and distance measurement for complex activity recognition.To improve model's descriptive ability,we design some spatio-temporal descriptors,integrating all spatio-temporal information in activity video,which eliminate the description gap and improve recognition accuracy.These methods compare the spatio-temporal distance between different activity videos by spatio-temporal description and distance measurement.These methods combine local features' structural information and capture their spatio-temporal relations,which can be used for both simple one-person action recognition and complex interaction recognition.(2)Hierarchical spatio-temporal probabilistic graphical model for complex activity recognition.To improve model's discrimination ability,we propose a hierarchical spatio-temporal probabilistic graphical model,directly modeling spatio-temporal relations and surrounding context in activity video,which eliminate the semantic gap and improve recognition accuracy.The goal of the method is to directly model local features'spatio-temporal relations with probabilistic graphical model and to train a model with strong discriminability from complex activity data.(3)Combine spatio-temporal context for activity recognition and localization.There are three major limitations in existing activity localization methods:a)To improve activity localization performance,we propose a novel spatio-temporal context model(STCM)to refine boundaries of target activity,which integrates various kinds of contexts.b)To narrow the activity search,we learn an optimal searching strategy by deep reinforcement learning instead of exhaustive searching,following human perception procedure.Experiment shows that the STCM obtains more accurate localization result without significantly increasing the number of proposals.c)To reduce the detailed frame-wise annotations,we propose a unified deep Q-network with weak reward and weak loss(WDQN)for human activity localization under sparse spatial supervision.It integrates some prior knowledge and underlying dynamic constraints into reward function and loss function to complement incomplete annotations.We evaluate proposed models on several public human activity datasets.The results show that spatio-temporal contextual information is not only helpful for activity recognition,but also contributes to activity localization.