The Research And Implementation Of Action Recognition Based On Zero Shot Learning

Facing to the diversity and richness of human actions in nature,the action recognition method based on supervised learning has been difficult to meet the needs of various applications.In recent years,the method of action recognition based on zero shot learning enables the model to complete the identification of new action categories without using corresponding samples for additional training,which greatly improves the generalization and universality of action recognition technology.In this thesis,we study the zero shot learning method via knowledge graph.In terms of the key problems and difficulties in action recognition,we propose a zero shot action recognition model based on graph convolutional network and action relation graph,which called GCNZSAR.The main research contents and work are as follows:(1)We propose using multi-modal fusion strategy to complete the design of the network structure,which contains three parts: video multi-modal feature extraction network,action category semantic encoding network and zero shot classifier.The first part is based on multi-stream framework,which can independently extract the visual features of video samples in various modals.The second part proposes using the graph convolutional network and action relation graph to encode the semantic vector of the action categories.This method can obtain better action category semantic features in multiple modals.The third part using a latent space to complete the zero shot classification,which fusion the multi-modal score of the action categories.This method alleviates the difficulty that existing methods cannot effectively utilize the multi-modal information of video.(2)According to the difficulty in constructing the action relation graph in the existing methods,we propose two new methods to obtain the association between action categories: one based on teacher network's confusion matrix and the other based on confusion matrix and meta-learning.These two methods are respectively aimed at whether there is a priori classification experience or not.They can effectively obtain efficient action relation graphs by using the visual difference of the action categories in multiple modals,which can improve the effect of GCNZSAR.The experimental results show that: GCNZSAR based on the action relation graph obtained by teacher confusion matrix has an average accuracy of 35.7% on UCF101 dataset and 25.8% on HMDB51 dataset,which reaches a state-of-the-art result.The zero shot action recognition system based on simplified GCNZSAR has a certain landing value on the embedded platform.It has an average recognition accuracy of more than 50% for five new action categories,as well as a prediction speed of 44.72 frames per second.The results achieve all the evaluation indicators.