Research On Fine Classification And Evaluation Of Human Action Based On Visual Data

Human action is one of the most important biological characteristics of humans,and it is of great significance for sensors and computers to perceive and understand human actions like human themselves.Human action classification and evaluation based on visual data has the advantages of non-contact,low cost,and easy access to data,which is widely used in the areas of intelligent security,human-computer interaction,and rehabilitation medicine.The fine classification and evaluation of human action based on visual data refers to using computer vision technology to predict the action category,times of occurrence,and score the action quality of human actions that occur in videos,which mainly includes action classification,action detection and action evaluation.The current research works mainly have the following problems:(1)Most of the existing action classification works rely on the deep learning frameworks,making it is difficult to conduct in-depth analysis and research on human action;(2)Current action detection algorithms do not consider the integrity of the action.The detection precision is low,and it is difficult to meet the needs of the actual scene;(3)The evaluation action is simple in the action evaluation research,and the application scene is limited to the evaluation of sports events.In response to the above problems,this dissertation starts from three aspects of action classification,action detection,and action evaluation.First,an interpretable action classification algorithm is proposed for category prediction of human actions that occur in trimmed videos.Considering that the videos captured in the actual scene are usually untrimmed long videos,a progressive action detection algorithm is further proposed to finely locate the human action in the untrimmed long video.Then,a fine action evaluation algorithm is proposed to quantitatively evaluate human movements with known categories and precise positioning.Finally,a digital visual-motor tracking system in the rehabilitation medical scene is designed,and the proposed action fine classification and evaluation algorithms are used to achieve an auxiliary diagnosis and mechanism exploration of correlation diseases.The main research works are summarized as follows.1.An interpretable human action classification algorithm based on the key-segment descriptor and temporal step matrix model is proposed.The action unit is defined as the skeleton segment composed of several consecutive frames with similar spatial structures.The spatiotemporal information of the segments is extracted and clustered to form a key-segment dictionary,thus the skeleton sequence is represented as a word sequence.A step matrix model is built to encode the multiscale temporal structure of the action sequence.The predicted class label of the test sample is the one whose action step matrix is most similar to the step matrix of the test sample.The recognition accuracies of the proposed method on the Northwestern-UCLA dataset,MSRC-12 dataset,and CAD-60 dataset are 78.96%,91.84%,and 91.18%,respectively.2.A progressive action detection algorithm based on deep residual network and action searching is proposed to achieve high-precision action detection.A progressive label is designed to quantify the action progress of the current frame.A 53-layer progressive label prediction network LPNet-53 is designed to implement progressive label regression of single-frame image.Then,a progressive action searching algorithm is proposed to output the interval locations where action instances occur based on the progressive label sequence.A dataset named DFMAD-70 is built to evaluate the proposed progressive action detection.The detection precision of the proposed method on the DFMAD-70 dataset is: m AP=97.0% at t Io U=0.5,m AP=76.0% at t Io U=0.8.3.In response to the needs of automated action evaluation in the rehabilitation medical scenario,an action detection algorithm based on a convolutional neural network with temporal filtering is proposed to realize fine action evaluation in the interactive process.The designed visual-motor tracking system is applied to record the action data of experimental subjects during the execution of fine action evaluation tasks,constructing an action evaluation dataset.Experiments show that the average error between the predicted result of the proposed action evaluation method and the manual score is 1.83,which means the proposed method realizes the effective automatic fine action evaluation.4.A digital markerless inexpensive visual-motor tracking system is developed,which is composed of Kinect and eye-tracker.Kinect is used to capture RGB images,depth images,and skeleton data that record the action information.The eye-tracker is used to capture the eye gaze information and the first-perspective action data.The designed system can be used to synchronous record action data and eye motion data in the process of action evaluation in rehabilitation medical scenarios,and to achieve an auxiliary diagnosis and mechanism exploration of movement disorders related diseases.