Multi-modal Commodity Identification System For Live Streaming Ecommerce

With the rapid development of short video platforms,the live video delivery industry is becoming more and more popular.The design and development of fine-grained multi-modal commodity identification system can greatly improve users’ experience and has practical significance.As the core module of the system,multi-modal named entity recognition can mine effective information in many multi-modal scenes,which has important research value.At present,the researchers of multi-modal named entity recognition mainly focus on feature extraction and the interaction and fusion of multi-modal information between text and image,but ignore the deep information of image and the influence of noise caused by irrelevant image.Therefore,in view of the shortcomings of the existing methods,this paper proposes two multi-modal named entity recognition methods,designs and develops a multi-modal commodity identification system in the live broadcast scenario with goods,and improves the effect of commodity identification based on the two proposed methods.The specific research content includes the following aspects:Firstly,this paper proposes a multi-modal named entity recognition method which introduces image attributes and background knowledge.InceptionV3 network is used to obtain image attributes,string matching algorithm and image similarity calculation method are used to obtain image background knowledge from multi-modal atlas,and a network model based on multi-dimensional attention mechanism is designed to fuse text,image attributes and background knowledge.Secondly,this paper proposes a multi-modal named entity recognition method based on graph-text correlation binary classification.Considering that the image will introduce too much noise for unrelated pairs of image and text,which will reduce the effect of named entity recognition.Therefore,a dataset of image and text correlation classification is constructed,and a unified framework is designed for image and text correlation classification and multimodal named entity recognition.Thirdly,we design and develop a multi-modal commodity identification system for live video streaming of sales.Based on the above two methods,the multi-modal commodity identification model is improved to enhance the effect of commodity identification,so that the research in this paper could be really implemented.Experiments show that the proposed two methods perform better than the multi-modal named entity recognition baselines on common datasets.And through the test,it can be found that the system designed and developed in this paper has a high accuracy and can meet the needs of users.