Research And Application Development Of Audiovisual Synesthesia Based On Emotional Semantic Matching

With the popularity of self-media,more and more short video producers have flooded into this trend.Choosing the right background music to embed in the captured video has become one of the most effective ways of video editing and atmosphere rendering.If the software can automatically select the background music that matches the image situation for the user,user satisfaction can be improved.Therefore,this article starts with audiovisual synesthesia,analyzes the emotional relevance of video and music from the perspective of human perception,and constructs a synesthetic perception model based on the similarity of video image-music in the emotional space,so as to recommend video Appropriate background music.The main research content of this topic is as follows: First,I will describe the research status of synesthesia theory,visual attention mechanism,deep learning and cross-modal retrieval methods,and understand their principles and development status.Subsequently,in view of the current situation where it is difficult to find bright spots in unedited short videos,an emotion-based audio-visual synesthesia retrieval model was constructed,that is,the use of deep learning methods to solve the mutual retrieval problem between two different modal data of video images and music And for the current emotion recognition direction that does not have a large-scale video image-music data set,this paper establishes a video image-music data set based on the hevner emotion model annotation.Then,through the audiovisual synesthesia mutual inspection experiment,the deep learning model was trained using the paired video-music pairs,and the synesthetic association between the video image and the music was established.The experimental results show that the audio-visual synesthesia retrieval model based on emotion in this paper can effectively measure the emotional similarity between video images and music.And through the user's evaluation of the synesthetic perception model,the feasibility of the model is verified again.Finally,based on kivy,a video image-music search APP was developed to help short video users match appropriate background music.