The Multisensory Integration Of Musical Emotions:from Behavior To Neural Substrate

Music is the language of emotion(Krumhansl,2002),which is the most common and typical art form for emotional communication.With the development of the times,music is no longer just an auditory art,but also a performance art combining audio and visual.The phenomenon of Multisensory Integration(MSI)of emotional information in musical performance and musical aesthetics is becoming more and more prominent.How to use visual emotional information reasonably in musical performance to obtain the best emotional contagion effect needs scientific guidance.The research on the mechanism of musical emotions based on audio-visual modality has received more and more attention.However,there are limited studies that use musical performance as audio-visual stimulus,and most research focuses on behavior results,with only a few studies exploring the neural mechanism.Meanwhile,complex and nuanced emotional information from visual and audio modalities have been the source of discrepancies between research results.Previous studies demonstrated varied results that might be caused by the following reasons: First,there are many different approaches to classifying and selecting emotions.Some studies use a dimensional(valence-arousal)approach,some use a discrete approach.Second,in terms of instrumentation,flute,clarinet,saxophone,piano,and drum have been used in previous studies.Third,studies varied in terms of duration,with exposure lasting 2 seconds,28 seconds,4 minutes,and so on.Fourth,some studies include participants with years of musical training experience while others include participants little or no musical expertise.Fifth,studies research on different orientations of musical emotions,some on emotion experience,some on emotion recognition.The present study used musical in AV modality as stimulus,used behavior and f MRI experiments,attempted to explore the SMI of musical emotions from behavior to neural substrate.The modality type,emotion type and musical training experience were taken as independent variables,based on the theory of music cue concordance.Reaction time,accuracy,intensity of emotion assessment and brain imaging results of music presentation phase and emotion assessment phase were taken as dependent variables.Focuses on the perspective of emotional congruence effect and emotional conflict effect,controversial topics in previous studies,and investigate the influence of emotion type and participants’ musical training experience that may lead to the difference between previous studies.On this basis,a theoretical model for the multisensory integration of musical emotions is proposed.Firstly,in order to control the additional influence of musical material characteristics,this study Preliminary Establish and Assess of Affective Music Video System,to provide rich,diverse and effective material sources for subsequent experiments.To avoid influential of musical features,materials included musical performances from the east and west,included opera,symphonic music,Chinese folk music and other art forms,played on different instruments.Durations of musical excerpts was about 30 seconds to make sure that participants received enough and constant type of emotional information(Hunter,Schellenberg,& Schimmack,2010).The selection of tracks is based on the material used in previous studies,the relationship between musical characteristics and induced emotion found by relevant studies,and the suggestions of music professionals.Assessment was conducted with 95 college students who evaluated the emotion type and intensity of all the clips.Materials were screened using a combination of strength,identification rate,item identification index and Chi-square test,and finally 64 pieces of music video which could induce target emotion(anger,fear,happiness and sadness)were screened out.The analysis shows that emotion recognition is more accurate and stronger than emotion experience.The emotional evaluation of happy music was the most accurate and uniform,followed by sadness,anger and fear.Participants had more accurate and stronger emotional assessment of Chinese folk music than western music.Secondly,explored the emotional congruence effect in MSI of musical emotions and its neural mechanism.In behavioral experiments,reaction time(RT),accuracy(ACC)and intensity of emotion evaluation indexes of musician and non-musician group in the Auditory only(AO),visual only(VO),and auditory-visual(AV)modalities were recorded.In experiment 2,music express happy and sad was used as materials.The results showed that the results of all emotion evaluation indexes in AV modality were better than that of VO modality,only RT and intensity of non-musician group in AV modality were better than that of AO modality.In order to investigate whether low evaluation difficulty of happy and sad lead to the result of experiment 2,and avoid the confusion between anger and fear leads to the too high difficulty,Experiment 3 adopts three emotion types of happiness,sadness and anger,and Experiment 4 adopts three emotion types of happiness,sadness and fear,carrying out with the same procedure as experiment 2.The results of experiment 3 also showed that,only RT and intensity of non-musician group in AV modality were better than that of AO modality.In experiment 4,none of the indexes showed better results in AV modality over the AO modality.But this effect is only stable compared with VO modality,and is only reflected in the emotion intensity of non-musician group compared with AO modality.These results suggest that adding visual emotional information congruent with auditory emotional information can improve the emotion recognition and emotion experience of musical performance in audiences without musical training experience.In f MRI experiment,BOLD signal music presentation and emotion assessment in AO and AV modality were recorded and analyzed,and music express happiness,sadness and fear,which had a great impact on the results of behavior experiment,were selected.The Middle Frontal Gyrus(MFG),Superior Temporal Gyrus(STG),Inferior Parietal Lobule(IPL),precuneus,Cingulated Gyrus,and Insula showed significant activation differences between modalities.Those brain regions overlapped with two brain functional networks,namely,the MSI of emotional information,and default mode network(DMN).This verifies that the MSI of musical emotions involving a large-scale dynamic network of multiple,interacting brain regions.And the MSI of emotional information,and DMN are important neural substrate for the MSI of musical emotion and emotional congruence effect.And,these results verify the influence of music emotion type and music training experience on mechanism of musical emotion processing.Thirdly,attempt to explore the emotional conflict effect in MSI of musical emotions and its neural mechanism,the third part of this study,compared the musical emotion assessment self-report results and brain imaging results in the congruous AV modality,incongruous AV modality,and AO modality,which answers the question whether adding visual information incongruent with auditory information can reduce the emotional contagious effect of musical performance,and which sensory,visual or auditory,plays a more important role in musical performance.There are two behavioral experiments,experiment 6 use the cross-modal matching paradigm,compare the results of incongruous AV modality and congruous AV modality,and experiment 6 compare the results of incongruous AV modality and AO modality,based on audio superiority and traditional ideas of music.The results showed that,the intensity of two group and ACC of non-musician group in incongruous AV modality was significantly lower than that of congruous AV modality.Intensity of musician group in incongruous AV modality was significantly lower than that of AO modality.In incongruous AV modality,ACC based on the emotion type of auditory sensory was significantly higher than that based on the emotion type of visual sensory.These results suggest that emotional conflict effect exists in musical performance,and the dominant sensory in conflict situation is auditory.Visual emotional information incongruent with auditory emotional information in musical performance can reduce the emotion recognition and emotion experience of musical performance,especially in audiences with musical training experience.In f MRI experiment,BOLD signal music presentation and emotion assessment in AO modality,congruous AV modality and incongruous AV modality were recorded and analyzed.The Superior Frontal Gyrus(SFG),Inferior Parietal Lobule(IPL),Cingulated Gyrus,and Thalamus showed significant activation differences between AO modality and incongruous AV modality.Those brain regions overlapped with brain functional networks of MSI of emotional information,DMN,and conflict control network.IPL,SFG,and precuneus showed higher activation in congruous AV modality than AO modality,while,IPL showed lower activation in incongruous AV modality than AO modality.These results suggest that the MSI of musical emotions in emotional conflict situation involving a large-scale dynamic network of multiple,interacting brain regions,the MSI of emotional information,DMN,and conflict control network are important neural substrate for the MSI of musical emotion and emotional conflict effect.And,these results verify the influence of music emotion type and music training experience on mechanism of musical emotion processing.According to the above experimental results,three conclusions are drawn in this study.1)The theoretical model for the multisensory integration of musical emotions.The degree of musical emotional congruent effect follows the principal of Inverse Effectiveness.The degree of musical emotional effect is positively correlated with the attention resource occupation,processing precision and cognitive load level of auditory information processing of musical emotions.2)The optimal mode for MSI of musical emotions is flexible.It is necessary to reduce or carefully use visual emotional information when performing sad music,and for audiences with musical training experience.Make full use of visual emotional information congruence with the auditory emotions when performing anger,fear and happy music,for audiences without musical training experience,and for activity and space limited enviorment.For audiences with musical training experience,the disadvantages outweigh the advantagest.3)The MSI of musical emotions is spontaneous processing with low cognitive load.DMN are important neural substrate for the MSI of musical emotion,the improvement of spontaneous self-reference processing is an important reason for the congruent effect of musical emotions.IPL is sensitive to musical emotional conflict.However,the core brain regions of emotional conflict processing did not show similar changes.The neural mechanism of musical emotional conflict effect is reflected in the increase of task difficulty,cognitive load level and attention resource investment,rather than inducing conflict processing.Both behavioral and brain imaging results reflect the uniqueness of the mechanism and neural basis of MSI of musical emotion information,which is different from non-musical emotions.This study used musical in AV modality as stimulus,and a theoretical model for the multisensory integration of musical emotions is proposed,through an innovative research perspective,expands the scope of musical emotion research which only regards music as sound stimulation,adapts to the current phenomenon of increasing use of visual information in musical performance,provides a guidance from psychological for the rational use of visual information in musical performance and musical education.Established affective music video,can provide emotional inducing materials for emotional regulation,psychological consultation,psychotherapy and mental health,and provide the basis and guidance for how to give full play to the role of emotional materials more efficiently.