Research On Chinese Song Emotional Classification Method Based On Multimodal Fusion

Music is an indispensable multimedia resource in daily life.The organization and retrieval of a large number of music works has attracted extensive attention of experts and scholars.Emotion is the most important semantic information of music.Classification based on emotion can effectively improve the efficiency of music retrieval,and has gradually become a research hotspot.There are three main problems in emotional classification of Chinese songs: 1)At present,there is no widely recognized Chinese music emotional dictionary,and most of the existing emotional classification methods of Chinese lyrics only consider the frequency of words,ignoring the impact of emotional intensity and part of speech on emotional classification.2)The low level descriptors of audio lack emotional relevance,and the high-level features are closer to emotions,but the cost of manual design is higher.3)Most of the related studies only use audio or lyrics to classify music emotions,while few studies combine audio and lyrics to classify multi-modal music emotions,and lack of public data sets.Aiming at the above problems,this paper studies the emotional classification method of Chinese songs based on multi-modal fusion,and carries out the following work separately:Firstly,in view of the fact that the existing emotional classification methods of Chinese lyrics do not make full use of the emotional information of music-related texts,a music emotional classification method for Chinese lyrics and comments is proposed.Firstly,a Chinese music emotional dictionary based on Word2 vec is constructed.The dictionary contains the emotional categories and weights of each word.Then,based on the dictionary,an emotion vector based on term frequency-inverse document frequency(TF-IDF)and part of speech is constructed,and finally music emotion classification is realized.The experimental results show that considering the influence of emotional intensity and part of speech comprehensively can effectively improve the classification effect,and the combination of lyrics and comments can further improve the classification performance.Secondly,the application of deep learning method in audio emotion recognition helps to bridge the semantic gap between low level descriptors(LLD)and high level emotional concept of music.Convolutional recurrent neural network(CRNN)is suitable for sequential data modeling.In this paper,CRNN is applied to music emotion classification,and a music emotion classification model based on LLD-CRNN is proposed.The model uses the spectrogram and low level descriptors of audio as input sequences to achieve complementary information.The input of CRNN is spectrogram,which extracts the time-domain,frequency-domain and sequence features of audio.The input of bi-directional gated recurrent unit(Bi-GRU)is LLD,which further obtains the sequence information of audio features.Finally,the feature of the two parts is connected to classify emotion.The experimental results show that LLD-CRNN model can effectively improve the classification effect.Finally,aiming at the deficiency that single modal data cannot fully express music emotion,a method of emotional classification of Chinese songs based on multi-modal fusion is proposed.Firstly,we construct text features based on emotional intensity and part of speech,and learn audio features based on LLD-CRNN model.Then,decision fusion and feature fusion are used for multi-modal fusion.The experimental results on the constructed Chinese song dataset show that using multi-modal information can effectively improve the classification performance compared with using only singlemodal information.