Fusion Of Skeletal And STIP-based Features For Action Recognition And Its Application

Human action recognition is an important topic in the field of computer vision, widely used in human–computer interaction, security surveillance, somatosensory games and other fields. In recent years, with the rapid development of depth sensing technology, action recognition based on depth information has become a hot research focus. How to accurately calculate the characteristic information is the key issue of human action recognition and has important research value. Action recognition technology based on fusion of skeletal and STIP-based features, and its application in human-computer interaction are studied in this paper.(1) This paper presents a view-independent skeletal feature to describe gesture, and proposes a STIP describtor to model the detail characteristics of human action. And a fusion scheme is proposed to combine the two features. Firstly, we use the Kinect sensor to get skeleton positions and depth image sequences respectively. By using median filtering and other operations, we improve the credibility of these information. For skeleton points, we build an independent local coordinate system and compute the deflection angle from them.For the depth image sequence, we detect STIP points and apply a custom descripor in the cubes around the STIPs. Secondly, HoG method is used to model the changes of skeleton features in the time-domain, as global feature. And we use Bag-of-words model to refine and simplify the STIP feature, as local feature. Thirdly, we reduce the dimension of the two features, normalized them, and then integrate these features. Finally, SVM is used to identify actions. Experiments show that the two features are complementary to each other,and the proposed method has better recognition results in complex scenes.(2) We design and implement an interactive application system based on human motion recognition. The system includes data acquisition module, continuous motion segmentation module, action recognition module and interaction module. The system can recognize actions, and implement human-computer interaction in Unity3 D virtual scene based on the recognition result. The system has a good interactive experience.