Audio-driven Dance Move Generation

As a key research content in the field of computer vision and cross-sequence analysis,music driven dance generation has been widely used in various life scenarios such as virtual reality,choreography,and animation production.At present,existing dance generation models either not finding a strong correlation between music and video or simply focusing on synthesizing human motion while ignoring the association between music and video.In this paper,we propose a novel model for synthesizing dance movements from music/audio sequence.This paper intensive studies the general process of dance generation,and then introduces Seq2Seq concept for dance generation.Moreover,this paper makes improvements for the current mainstream learning to Seq2Seq methods according to the characteristics of dance generation and introduces a set of evaluation criterion for synthetization,which do not have source for reference.The major innovative works are as follows:(1)A cross-domain Seq2Seq learning framework is proposed for realistic dance generation.While various Seq2Seq methods have been proposed,most of them only demonstrate their effectiveness on machine translation.Considering the characteristics of the music and dance,in this paper,we improve the current mainstream Seq2Seq method and propose a new cross-domain sequence analysis method called Long Short-Term Memory and Self-Attention(LSTM-SA)model,in which the learned model can generate notable and natural dance sequence.(2)A set of evaluation criterion was proposed for synthetization evaluation,which do not have source for reference.There is no clear reference mapping between music and dance movements,so it is necessary to find some criteria to evaluate whether the generated dance sequence was natural and whether it was produced in accordance with the music.Considering that there are few studies on dance sequences,this paper sets some new rules for judging the advantages and disadvantages of the generated model.In order to ensure the comprehensiveness of the evaluation,this paper is evaluated from two aspects: subjective evaluation and objective evaluation.Specially,this paper proposes a manual scoring,creates a new learning based scoring model,and calculates the correlation coefficient between the original dance sequence and the generated dance sequence.(3)A dance dataset that including both music and corresponding dance motions was collected.Few motion capture datasets include music & dance movement data.To our knowledge,no dataset of synchronized music and motion capture data is currently available online.In order to realize the generation model of music-driven dance movements,we have created our dataset.The dataset is a relatively high quality of dance video collected from the internet.(4)This paper implemented a music-driven dance generation system.The system integrates the music-driven dance generation process into three major modules: the selection of music samples,the synthesis of dance sequences,and scoring of synthesized dance sequences.With this system,the results of dance generation in different generation stages of the music sequence can clearly displayed,and make a comparative analysis of the performance of dance generation in music based on different Seq2Seq models.