Research On Action Recognition Based On Bimodule And Multi-Feature Fusion

Action recognition(AR)is an important research subject in computer vision,which means to use the computer vision technology to describe and recognize the actions in video.In recent years,new technologies and methods have been applied in this task,but most of them are based on visible imaging videos,which leading to the largo and immature development of infrared action recognition.But in some situations which require monitoring around the clock,visible and infrared data can complement each other very well,and even no one is dispensable.In view of this situation,this thesis focuses on the research of action recognition based on visible light and infrared dual mode data,the detailed work are listed as follows:First of all,a bi-module action recognition dataset is constructed based on a survey of large numbers of exiting action recognition datasets.The dataset consists of 12 daily action types,each spectral type has 50 video clips under one action type.Totally 1200 video clips.We also take into account many imaging factors including background,viewpoint,occlusion,disruptors and so on when shooting samples.Then several popular action recognition algorithms are used to evaluate this dataset.Then,an infrared action recognition algorithm based on the fusion of weighted dense trajectory and deep learned features is proposed,which is tested on the infrared part of the bi-module dataset.According to the characteristics of infrared data that imaging relay on the target's infrared radiation,this algorithm improves the dense trajectory descriptor by adding gray value weight into it,then the feature extracted from the interest point of the body target will be endowed with a higher weight.After that the improved descriptor is fused with the deep learned feature by decision level fusion,finally,the recognition accuracy of 72.97 percent is obtained in the infrared part of the bi-module dataset.Last but not least,based on the above work,a two-stream action recognition method based on bi-module and multi-feature fusion is proposed.A channel processes the visible light video clips while the other one processes the infrared video clips.According to their different characteristics,for the visible data channel,the full connect layer feature of optical flow convolutional neural network and the spatial pyramid pooling feature are extracted,for the infrared channel,the contour information of motion history image and weighted histograms of oriented optical flow are extracted.That means diverse features are extracted each channel to express the appearance and motion information of the action to be identified.Then the identify result are given by the adaptive fusion model.Experimental results show that the proposed algorithm can get higher recognition accuracy than the state of the art in the bi-module action recognition dataset.