Research On Multi-dialect Speech Recognition And Multi-modal Sentiment Analysis Algorithm Based On Knowledge Transfer

Information is a bridge that connects people,as well as people and computers.Due to the rapid development of human-computer interaction technology in recent years,the interaction between humans and computers has become possible,but compared with human-human interaction,there are still many problems in human-computer interaction.The performance is poor in multi-dialect scenarios,and it is difficult for sentiment analysis models to mine the information correlation between modalities on multi-modal data.Based on human experience in learning various knowledge,the model can quickly understand and absorb new knowledge by using the knowledge and ability learned in the past.Therefore,using the method of knowledge transfer can improve the model’s ability to learn and understand knowledge.This thesis proposes the research of multi-dialect speech recognition and multimodal sentiment analysis algorithm based on knowledge transfer.Aiming at the difficulty of dialect speech recognition in the multi-dialect speech recognition tasks in practical scenarios and the existence of multi-modal sentiment analysis tasks are difficult to extract emotional information and difficult to synergize modal features,the application of the theory and method of knowledge transfer to multi-dialect speech recognition tasks and multi-modal sentiment analysis tasks to study knowledge transfer methods in different scenarios.Aiming at the problem that there are many kinds of dialect data and the number of each kind of data is small in multi-dialect speech recognition task,a multi-dialect speech recognition algorithm based on meta-transfer learning is proposed.First,we study the current mainstream knowledge transfer algorithms,and these algorithms are used in the field of multi-dialect speech recognition to explore the effectiveness of knowledge transfer methods.Second,from the perspective of knowledge transfer,the multi-dialect speech recognition algorithm based on meta-transfer learning combines the advantages of transfer learning and meta-learning,and realizes the transfer of knowledge as well as the transfer of learning ability,and the gradient analysis proves that our algorithm can speed up loss optimization while optimizing the expected loss.The experimental results show that this method can learn how to learn dialects in the process of learning multiple dialects when there are many kinds of dialects and few data for each dialect,and obtain knowledge of specific dialects in specific dialect materials.Compared with the current state-of-the-art method,this method achieves a7.8%～10.9% drop in word error rate on different dialect data.Aiming at the problem that traditional models have weak ability to represent modal features and cannot effectively mine information correlation between modalities in multimodal sentiment analysis tasks,a multitask sentiment analysis algorithm based on modal features transfer and fusion is proposed.First,we extract modal features through large-scale pretraining models of different modalities,and realize the knowledge transfer from pre-training models to sentiment analysis tasks.Second,we design a multi-scale intra-modal and intermodal feature fusion network,which achieves intra-modal feature enhancement and intermodal feature fusion,and realizes information complementarity and knowledge transfer between modalities.Next,a multi-task learning architecture is constructed.In addition to the main task of sentiment analysis,the multi-modal emotional feature enhancement auxiliary task and the inter-modal emotional feature cooperation auxiliary task are designed to realize knowledge sharing between the main task and auxiliary tasks.Finally,the algorithm is tested on the CMU-MOSI and CMU-MOSEI datasets.The experimental results show that our method can achieve 86.8% and 87.5% accuracy in the binary classification of the sentiment analysis task,an increase of 2.5% and 1.2% compared with the state-of-the-art method,respectively.The practical application results of knowledge transfer method in the multi-dialect speech recognition task and the multi-modal sentiment analysis task show that knowledge transfer method can transfer and learn by using the knowledge and ability learned by model,and has excellent performance,and can be widely applied to various task scenarios.