Research On Personalized Affective Interaction Based On Multimodal Semantic Analysis

Affective interaction technology based on pattern recognition and cognitive theory has been widely used in fields such as medical care,elderly companionship,and intelligent education.It is an indispensable part of the national artificial intelligence development blueprint.Due to the diversity of data sources in the Internet of Things,which greatly increases the complexity of data distribution in the affective interaction,the effectiveness of affective interaction faces challenges.Multimodal signals based on peripheral physiological signals and facial expressions can more comprehensively reflect the emotional state of humans.However,the multiple distribution forms of modal data increase the difficulty of feature processing,which limits the effectiveness of multi-modal emotion recognition.The existence of personalized differences causes that the emotional perception capabilities of smart devices is limited,and behavior feedback cannot meet the needs of users.Therefore,in view of the deficiencies of the existing research work,this paper extracts the potential emotional information of multimodal features,proposes an adaptive emotion recognition model,and verifies its effectiveness of emotion recognition.Then,a model structure based on cognitive analysis and emotion regulation is designed to simulate the process of human emotion,cognition,and behavioral decision-making.Specifically,this paper focuses on the research of personalized affective interaction based on multimodal semantic analysis.Based on the analysis of the shortcomings of existing work,the multimodal feature processing method combined with emotional semantic analysis,the adaptive emotion recognition model based on transduction learning,and the emotion-cognition collaboration model are respectively proposed.The main research contents and innovations of this article are as follows:(1)A multimodal feature processing method combined with emotional semantic analysis is proposed.In view of the high computational cost and dimensionality curse of traditional methods,the multimodal feature processing method combined with emotional semantic analysis is provided,to improve the ability of multimodal features to represent emotional semantics.Use the form of 3-dimensional convolution to process the temporal and spatial information of expressions based on continuous video to extract the effective features of expressions;provide a time-frequency-based peripheral physiological feature extraction method,and use deep belief networks to extract its nonlinear high-dimensional features;use emotional semantic analysis based on embedding layer,to measure the correlation of emotional semantics between modal features;use the method combined with the logistic regression to realize the multimodal feature fusion based on the soft attention mechanism.Experimental results based on the RECOLA multimodal public database show that the feature fusion method based on semantic analysis can get the best recognition result in the provided work.It indicats that the feature processing method combined with emotion semantic analysis can use semantic information to improve the effect of multimodal emotion recognition.(2)An adaptive emotion recognition model is established based on transduction learning.For the inability to learn the distribution of specific instances using inductive learning in existing studies,a method of constructing a recognition model based on transduction learning theory is proposed to reduce the impact of uneven distribution within classes caused by personalized differences.Considering semantic correlation,the aggregation degree of the feature distribution and label distribution in the training set is taken as one of the optimization goals to establish a transduction learning model in the high-dimensional feature space;the objective function of the support vector regression is optimized,and the transduction learning is integrated into the model training process;the experiments based on the multimodal public database and the acquisition platform based on wristbands are provided.The experimental results show that the multimodal emotion recognition results of the adaptive emotion recognition model based on transduction learning is better than the models based on inductive learning,and it improves the generalization of the model.(3)An emotion-cognition collaboration model is proposed.For the problem that existing emotion modeling research did not consider about the time dimension of cognition-emotion collaboration,a model structure based on cognitive analysis and emotion regulation is designed,and a bottom-up collaboration method based on different preferences is proposed,which realizes emotional state transfer and behavioral decision-making.Under the influence of external stimulus,a quantitative motivation generation framework is established based on the characteristics of competitive neurons;use autoregressive time series to realize the process of emotional state transfer,and introduce the Gross emotion regulation theory to realize the cognitive reappraisal strategy;use the "exploration-only"strategy in reinforcement learning to measure the influence of emotional state and motivation on behavioral decision-making.The statistical results based on interaction experiments show that the robot based on the emotion-cognition collaboration model can achieve different effects on behavioral decision-making and emotional state transfer by combining different preference settings under the stimulation of user emotional feedback and interactive behavior.It realizes the natural emotional state transfer process under continuous external stimulus.And the results verify the effectiveness of the cognitive reappraisal strategy.(4)A multimodal personalized affective interactive learning platform is established.For the problem of the lack of unified verification of emotion recognition and emotion modeling in existing affective interactive platform research,a design architecture of the multimodal personalized affective interactive learning platform is proposed.It includes the sensor data collection process and the above mentioned affective interaction theory process.The complete human-computer interaction experiment process is introduced,and the personalized affective interaction learning platform is analized in two parts:emotion recognition and affective human-computer interaction.The best average Concordance Correlotlon Coefficient result of the adaptive emotion recognition model based on multimodal semantic analysis is 0.437,which is higher than the related work provided.The affective interaction results based on statistical analysis show that the platform can generate behaviors based on cognition-emotion modulation that meet the teaching expectations,according to the recognized user emotional state and interactive behaviors during the interaction process.It achieves more positive teaching interaction,and verifies the effectiveness of personalized affective interaction based on multimodal emotional semantic analysis.