Research On Automatic Labanotation Generation Based On Time Series Analysis Of Human Motion

Traditional folk dances are an important part of the local intangible cultural heritages.Recording traditional folk dances with Labanotation scores is a good means of protecting the intangible cultural heritages.Labanotation is one of the most widely used notation systems to record human dance movements in the preservation,reproduction,education and communication of dances.However,most of Labanotation scores are written by hand,which requires a huge amount of time and effort to observing,drawing and proofreading even for professionals.Therefore,using computer technology to automatically generate Labanotation scores is of great value in application and research.At present,the mainstream methods of automatic Labanotation generation first obtain human motion data via motion capture technologies,and then generate Labanotation scores based on analyzing and recognizing the captured motion data.This task takes the human motion data sequence that varies along time as inputs,and outputs Laban symbol sequences along the time order.Therefore,this dissertation investigates temporal sequential analysis methods,focuses on the research of antomatically generating Labanotation scores based on human motion data,proposes a number of specialized motion feature extraction algorithms and temporal analyzing models,and significantly improves the accuracy of Labanotation generation.The main research work can be summarized as follows:(1)We propose a method that automatically generates Labanotation scores based on data segmentation.For manually segmented data,in order to overcome the drawback of template matching methods that is not flexible enough,we propose a method based on time series analysis with hidden Markov model for the automatic Labanotation generation.First,in order to deal with the challenges including various dance forms,different dances’ body shapes and noises in motion capture data,we propose a new feature,which is invariant to human body measurement and body orientation.Then,we apply hidden Markov model to analyze the temporal dynamic characteristics of limb movements and map each limb movement to the corresponding Laban symbol.Furthermore,in order to save manpower,we propose an automatic generation framework of Labanotation scores based on fully automatic data segmentation.First,according to the center of body gravity transferring theory of Labanotation,the continuous motion capture data are divided into data segments each containing only one movement.Then,we use a neural network with one-dimensional convolution layer and recurrent layer to recognize the data segments and obtain the corresponding Laban symbols.(2)We propose a method that generates Labanotation scores automatically based on convolutional recurrent attention sequence model.In order to eliminate the influence of erroneous data segmentation on the automatic Labanotation generation,we propose an attention sequence learning model based on convolutional recurrent networks with fusion features to generate reliable Labanotation scores directly from continuous temporal sequences of motion data.First,we fuse the bone feature and Lie group feature,so that the fusion features can not only extract the bone information between adjacent joints,but also learn the relative geometric relationships between connected bones.Then,in the sequence learning model,we use convolutional recurrent networks to learn the spatio-temporal representation from motion capture data and employ an attention mechanism to learn a good alignment between the input motion feature sequence and the output symbol sequence.Finally,the correct Laban symbol sequences and Labanotation scores are generated.(3)We propose a method that generates Labanotation scores automatically based on graph convolutional networks and attention sequence learning model.In order to fully exploit the motion feature information from skeleton data,we propose a new attention sequence learning model based on graph convolution to analyze time series of human motion for the reliable and efficient automatic Labanotation generation.In the encoder,we propose a new gesture-sensitive graph convolutional network with learnable adaptive joint weights and non-physical connections to learn both spatial and temporal patterns from motion data sequences.In the decoder,we exploit motion rhythm information and propose a novel rhythm-aware attention mechanism to learn a good alignment between motion sequences and Laban symbol sequences,so that we can focus on relevant parts of the input motion sequence without searching in the whole input sequence when predicting a target Laban symbol.Therefore,we can generate Labanotation scores efficiently and accurately.(4)We propose a LabanFormer model that generates Labanotation scores automatically based on a graph attention network and the Transformer model.In order to effectively capture flexible limb movements and deal with the temporal sequences of complex dance steps,we propose a LabanFormer model based on graph attention network.First,we propose a multi-scale graph attention network(MS-GAT)that can capture flexible limb movements by learning feature correlations between every two joints and aggregating features of neighboring joints over multiple scales.Second,we propose a new Transformer model with a gated recurrent positional encoding(GRPE)module to learn the global temporal dependencies in the output feature sequences of MS-GAT.The novel GRPE module can encode position information with learnable parameters while handling time series of various lengths.As such,the periodic,symmetric,or repeated steps in dances can be effectively captured.After training,the proposed model can accurately decode motion capture data sequences and generate corresponding Laban symbols.We carried out sufficient experiments on two real-world datasets.A large number of experimental results show that the automatic Labanotation generation algorithms based on human motion time series analysis proposed in this dissertation can obtain favorable generation performance.The accuracy is progressively improved with the proposal of each method and the proposed algorithms perform favorably comparing with the stateof-the-arts.Therefore,we can contribute to the process of folk-dance protection.