Product Information Matching System Based On Image Recognition And Text Classification

In recent years,as the scale of e-commerce platform continues to expand,it has become a huge task to judge whether merchandise images and text descriptions uploaded by business are matched to audit information.Nowadays,audits are generally conducted manually.There is a low degree of automation and a large amount of manpower and resources are involved.At the same time,real-time audits can not be guaranteed.In addition,the commodity information in the e-commerce platform also has the characteristics of fine-grain classification,it is difficult to distinguish the slight differences between the product categories by means of manual auditing,which increases the difficulty of the auditing work.Therefore,it is a widely used research problem that how to effectively annotate the commodity image and text description,judge whether the image text matches,and complete the audit process automatically and in real time.Starting from the actual demand,this paper proposes a method of product image and text description matching based on convolutional neural network and long short term memory network.By using the feature extraction ability of the deep network model,images and texts are annotated to solve the fine-grain classification problem that traditional training methods can not achieve.And make sure the task of auditing commodities information is completed automatically.Firstly,propose an image and text matching framework based on deep neural network.This method differs from traditional image or text annotation methods which separate the feature extraction and classifier training into two independent steps,by using the feature extraction capabilities of deep neural network so that end-to-end training is accomplished to achieve fine-grained classification goal,and finally through the label comparison to achieve the image text matching process.Secondly,design an automatic image recognition method,which is based on convolutional neural network model.In order to resolve the conflict between a large number of training samples are needed in model training and limited labeled data,using the data augmentation method like flip to generate new image to complete the expansion of the data set.At the same time,for most of the images can be separated into two parts,that is the target object area and the background area.In order to effectively improve the accuracy,only the part containing the target object after region segmentation will be used to model training process.Finally,images after data augmentation and region segmentation are used to train the convolution neural network.Thirdly,design an automatic text classification method,which is based on long short term memory network model.In view of the traditional method which treat words as independent individuals without considering the semantic relations between words,word2 vec is used to convert words into distributed word vectors.At the same time,TextRank method is used to extract key words from labeled documents,and the word vectors corresponding to these keywords are spliced into a matrix to be used as the input of long short term memory network to train model for text annotation.Fourthly,a prototype system based on the image and text annotation methods is designed and implemented.The prototype system uses Java to develop,Caffe and TensorFlow for model training.The use of TFS is for storing images and text files,and uses MySQL to achieve structured data persistence.The proposed framework for image and text matching based on deep network model effectively solves the actual needs.And the design,implementation and verification of the prototype system verify the effectiveness of the proposed method.