Automatic Labanotation Generation Of Continuous Movement Based On Deep Learning

With the rapid development of computer technology and artificial intelligence,employing digital and intelligent means to protect and inherit intangible cultural resources has become an important research topic.Labanotation is a scientific analysis and recording system of human movements,which has been widely applied and disseminated all over the world.As a form of written record easy to store,it plays an essential role in recording and preserving traditional dances.Since hand-drawing notation is a complicated task,automatic Labanotation generation using computer technology came into being.However,algorithms under the traditional Labanotation generation framework rely on pre-segmentation of motions,and cannot be globally optimized.In addition,the commonly-applied skeleton features cause certain information loss,the spatiotemporal modelling capability of the existing recognition models need to be improved.Therefore,on the basis of enhancing skeleton representation and spatiotemporal modelling,this paper mainly investigates on the end-to-end deep learning framework for continuous Labanotation generation,and proposes four algorithms,which are summarized as follows:(1)A Labanotation generation algorithm based on two-stream spatiotemporal parallel model.The Lie group feature of motion capture data is designed for capturing the rotation information.It represents movements as a high-dimensional trajectory on the Lie group,composed of rotation matrixes between adjacent joints and adjacent bones,and the matrix multiplication operation.In terms of spatiotemporal modelling,the parallel model using LieNet and the long short-term memory networks(LSTM)is proposed.It separately processes the Lie group and the bone vector features,fuses the two streams by the score fusion.In this way,it combines the spatial modeling of LieNet with the temporal modelling of LSTM,and achieves better recognition performance than single models.(2)A continuous Labanotation generation algorithm based on spatiotemporal series model.An end-to-end framework based on the connectionist temporal classification(CTC)is proposed for directly generating continuous Labanotation.The Lie group feature is processed by the convolutional neural networks(CNN)followed by bi-directional gated recurrent neural network(Bi-GRU)to achieve spatiotemporal modelling.The CTC trains the entire model in an end-to-end manner to output discriminative fined-grained descriptions of gesture in each frame,and uses them to solve out the optimal symbol sequence,so that the model can flexibly recognize motions with different durations.This method effectively avoids the labor of manual segmentation,reduces the system complexity and improves the recognition accuracy.(3)A continuous Labanotation generation algorithm using double-stream directed neural networks(DGNN).An orientation-normalized double-stream directed graph feature for motion capture data is designed to effectively represent the kinematic dependencies underlying in the skeleton.The double-stream fusion framework of DGNNs(DFGNN)is proposed,which processes the double-stream feature with two branches,fuses them with a fusion-pooling module and solve out the symbol sequences with the CTC.Experiments prove that the proposed method achieves high-quality Labanotation generation performance.(4)A refined continuous Labanotation algorithm based on semi-supervised dynamic frame clustering.On the basis of continuous recognition framework in(3),a semi-supervised dynamic clustering module is added.The dynamic clustering based on k-means is employed to roughly cluster each frame,then the smoothing and pooling of clusters is performed to accurately label the duration of motions,with the motion number obtained by the recognition model as prior information.In this way,the refined generation of continuous Labanotation is achieved.