| Video modalities describe high-dimensional objects and their actions through frame sequence structures,and their complex dependencies in spatial and temporal dimensions bring challenges to recognition tasks,especially the fine-grained recognition task.Aiming at the difficulty that high-dimensional video has small inter-class differences and large intra-class differences in the fine-grained action recognition task,this thesis designs a focus and fusion mechanism to extract low dimensional features of frames in spatial dimension and frame sequences in temporal dimension.Specifically,this thesis takes the static visual information within frames and the optical flow summary information representing action information in the temporal dimension as the objects.Then this thesis adopts a multi-semantic selection mechanism to capture the finegrained features in the spatiotemporal dimension separately,and dynamically fusions the twodimensional multichannel features with the aid of the attention mechanism.This thesis implements the end-to-end prediction of the model by generating continuous features from stillness to motion according to the expansion of frame features at certain moments in the temporal dimension.The specific work is as follows:1)Multi-semantic fine-grained feature extraction of focus mechanism: Propose a finegrained feature extraction network based on the focus mechanism.Use the multi-head attention mechanism to work on different regions of feature maps and feature spaces of different abstraction levels.Integrate the original attention weights of each layer into one attention map and select the most discriminative image region under each semantic as the objects for fine-grained feature extraction.Enlarge the distance between similar subclass features.Enhance the model’s ability to locate discriminative regions and extract fine-grained features.2)Multi-channel attentional spatiotemporal information fusion: Propose two kinds of multi-channel spatiotemporal information dynamic fusion networks based on attention mechanism.Use the multi-head attention mechanism to learn the spatiotemporal information’s fusion coefficients dynamically,and improve the branch fusion and the rule fusion methods of temporal and spatial streams.According to the action information sampled from different sub-segments,dynamically weight feature summaries of each branch based on visual or motion features of different segments,and dynamically weight rule summaries of segment consensus features based on the splicing of visual and motion features of each segment.At last,improve the utilization of action information.3)Fine-grained action information generation: In response to the problems of huge computation and memory occupation of traditional optical flow methods,propose a fine-grained action information generation network based on action contours.According to the visual feature encoding of adjacent images and the RGB difference matrix,action features oriented to the motion target contour are generated to reduce the dependence of the recognition task on background information and improve the efficiency of fine-grained feature extraction.Add the action information generation module to the temporal neural network branch to implement endto-end prediction of the fine-grained recognition model.Carry out experiments on the datasets Fine Gym and MLB-Youtube to verify the effectiveness of focus and fusion mechanism for action fine-grained feature extraction and verify the effectiveness of generating optical flow for action feature representation. |
