Research And Application Of Multi-modal Feature Learning Algorithms Based On Graph Network

Multi-modal data is a human abstraction of different forms of things in nature,and multimodal data is often the carrier of people's information exchange.Therefore,how to perform feature learning on multi-modal data is an important basis for data recognition and understanding.Due to the rapid development of deep learning in recent years,the feature extraction of deep learning on single-modal data has been improved,but compared to the learning process of humans,there are still many problems in deep learning.For example,deep learning requires a large amount of labeled data for training,and the inability to introduce human prior knowledge for reasoning and migration.According to the experience of human learning knowledge,the topology can be used to construct a knowledge system intuitively.Graph networks are very suitable for both multi-modal feature extraction and multi-modal feature fusion.There,this thesis proposes a multi-modal feature learning framework based on graph networks and designs different multi-modal feature learning algorithm modules based on problems encountered in different multi-modal learning tasks.This thesis will focus on the problems encountered in traditional deep learning in object detection and recommendation systems,and study multi-modal feature learning algorithms based on graph networks.Aiming at the problem that the feature distribution of target detection model training samples and actual test samples in existing mobile traffic scenarios is different,a knowledgesharing framework for intelligent transportation mobile edge computing is proposed,and multi-modal features are used to realize knowledge sharing between edge computing nodes.To overcome the problem that traditional object detection algorithms cannot establish relationships with other modalities through spatiotemporal relationships so that multi-modal data cannot be used to implement knowledge transfer in edge computing scenarios,this thesis proposes the Cross-Modal Contextual Knowledge Reasoning(CMKR)multi-objective detection algorithm module.This algorithm uses graph networks to extract relational features in images,and that uses multi-modal bilinear pooling to fuse cross-modal contextual knowledge with relational features between image instances and infer.Then,by simulating the training process of the model in mobile edge computing,the effectiveness of the CMKR module in cross-domain object detection tasks in traffic scenes is verified,and the algorithm module improves the ability of small sample transfer learning under the knowledge sharing framework,and the detection accuracy is improved by up to 4%.Traditional recommendation systems cannot make full use of the features extracted from multi-modal data,and therefore cannot well solve the problems of data sparseness and cold start in the recommendation system.Therefore,according to the characteristics of the recommendation system tasks and the topology of the knowledge graph,this thesis proposes an algorithm named Knowledge Graph Multi-Relational Collaborative Filtering(KGMRCF),which combines a graph convolutional network with a relational graph convolutional network.In addition,it uses multi-Layer heterogeneous graph feature fusion network to fully learn the collaborative relationship between the knowledge graph and the user,and the model can set different relational graph convolution layers to adapt to different recommended data structures and requirements.Extensive experiments show that performance improvement of the KGMRCF on some standard datasets can reach 2 to 5 percentage points,and the model training speed has reached 3 to 4 times.The practical application results of multi-modal feature learning based on graph networks in object detection and recommendation systems show that graph networks have excellent performance in constructing feature relationships of different modalities and using external prior knowledge for migration and inference.The multi-modal feature learning framework of the network can also be widely used in other multi-modal application scenarios.