The Key Technology And Experimental Research Of Multi- Mode Interactive Music Perception

Music is the language and art of emotion, and emotion is the essential feature of music. But so far,people perceive music by hearing,without an effective method to see or touch music,which mean that people can't experience interactively music emotion by multiple perception modes through audio-visual-tactile stimulation. In this study, based on integrating multi-disciplinary knowledges of musicology, cognitive psychology, psychophysics, multimedia technology, signal processing, pattern recognition and intelligent control, virtual reality and instrumental science, the extraction and recognition technologies of music emotion characteristics were studied to improve the recognition rate of music emotion and to enhance immersion when enjoying music. The thermal tactile perception and reproduction experiment device (TTPRED) and visual music perception experiment system (VMPES) were designed. A series of psychophysical perception experiments were carried out systematically, including the human finger thermal tactile perception experiment (HFTTPE), visual music perception experiment (MPE) based on audio-visual stimulation, MPE based on audio- thermal tactile stimulation, multimodal MPE based on audio-visual-tactile stimulation, and so on.The main innovation of this thesis is using thermal tactile to perceive music. The expression form of visual music which can produce strong immersion, the design scheme of music perception using thermal tactile, and the design scheme of multi-channel interactive music perception based on audio-visual-tactile stimulation were proposed from the technical level by carrying out a series of experiments of psychology and physics. Through scientific research of the project, it provides the theoretical basis and technical support for the design of music perception interface equipment which can be used to enjoying music based on audio-visual-tactile stimulation.The main research works and progresses of the paper are as follows:(1) In this paper, music emotions are modeled as a set of continuous variables composed of valence and arousal (VA) values based on the Valence-Arousal model. Music features, including energy,rhythm, harmony, temporal, and spectrum feature, were extracted and selected by method of principal component analysis space projection and Relief feature selection. A wide range of methods including multivariate adaptive regression spline,support vector regression (SVR), radial basis function, random forest regression (RFR), and regression neural networks were adopted to recognize music emotions. Experimental results show that these regression algorithms have led to good regression effect for music emotion recognition (MER). The optimal R2 statistics of VA values are 29.3% and 62.5%,respectively,which are obtained by the RFR and SVR algorithms in the relief feature space and are superior to the results reported in the literatures. A wide range of methods including support vector machine classification, K neighborhood classification, fuzzy neural network classification, fuzzy K neighborhood classification, Bayesian classification and Fisher linear discriminant algorithm were adopted to classify music emotions. Among them, based on the support vector machine, fuzzy K neighborhood and Fisher linear discriminant algorithm, the correct rate of music emotion classification (MEC) are greater than 80%. A new hybrid classifier algorithm was proposed, and MEC rate is as high as 84.9% using this new algorithm.(2) Using thermal tactile sensing mechanism based on semi-infinite body model, embedded technology,fuzzy control technology and virtual instrument technology, three TTPREDs were designed. The temperature control range of TTPREDs is from-10? to 130?, the temperature resolution and precision are 0.01? and ±0.1? respectively, and the maximum heating or cooling rate is greater than 12?. A series of experiments, including The HFTTPE, difference threshold measurement experiment of thermal tactile, experiment of mapping relation between emotion and temperature, emotion recognition experiment based on thermal tactile stimulation, and Virtual material identification experiment based on thermal tactile, were carried out respectively.Temperature threshold value of finger thermal sensation and comfort perception and finger thermal tactile difference threshold were measured. It is verified that the human emotion and thermal sensation are consistent and harmonious by the measurement of the thermal sensation and comfort,and emotional awareness can be enhanced through the thermal tactile sensing channel. The experimental results further show that thermal tactile perception and rendering technology can strengthen the sense of reality and immersive in the cognitive process and emotional experience.(3) A novel VMPES was developed based on HTML5. The VMPES is easy to operate and can act in a variety of performance forms, which can meet the requirement of multi-mode music perception experiment. Using VMPES, through questionnaires based on the autonomous perception experiment and statistical analysis, the different emotional types of visual music performance forms were determined, which provides design schemes for developing visual music synchronized with musical emotion expression.(4) According to the psychophysics experiment criterion, the experimental scheme of four factors mixed design with two factors of repeated measurement and the experimental scheme of the two factors completely randomized experiment design were selected, and a series of psychophysical perception experiments, including visual MPE based on audio-visual stimulation, MPE based on audio- thermal tactile stimulation, and multimodal MPE based on audio-visual-tactile stimulation,were carried out systematically using VMPES and TTPRED. Based on statistical analysis of experimental data by SPSS software, the main effects of music emotion type, dynamic image emotion type of visual music, and thermal tactile emotion type on the immersion degree of the musical emotion were tested respectively. Meanwhile, the main effects and simple effects of interaction between each other were tested. Then, the effective design scheme of multi-channel interactive music perception based on audio-visual-tactile synchronous stimulation was proposed.