Study On Event Image Classification By Fusing Multiple CNNs Based On LSTM

The development of mobile Internet,mobile apps,and social platforms has brought massive amounts of image information.Images have become the main medium for the exchange of information on the Internet.Compared with other massive images,event images covers complex visual information and special semantic content.Although it brings a quick way to record and share information,it increases the difficulty of content analysis and reduces the efficiency of information retrieval.Event image classification technology is an important way to solve this problem.However,visual information such as scene environment,position of an object and human gesture in an event image is difficult to represent.It is still difficult to correctly identify the image category.For the problem of event image classification,this paper proposes an event image classification method based on LSTM fusion multi-CNN to effectively classify event images.The research method of this paper has important theoretical significance and practical application value.The main work of the paper is as follows.1.The feature extraction method of event image based on convolutional neural network is studied.It fully considers the characteristics of events in the target database and fuses a variety of visual features to represent the event.We obtain the feature extraction network of object by fine-tuning the VGG 16-Net network model pre-trained on the ImageNet dataset.The feature extraction network of scene is obtained by fine-tuning the VGG16-Net network model pre-trained on the dataset Places205,and a feature extraction network of character is obtained by fine-tuning the AlexNet network model pre-trained on the PIPA dataset.Experiments show that the extracted visual features of scene,object and character can be used as a reliable basis to classify event images.2.The fusion method of feature information is studied.Through a large number of experiments,we have verified that three different visual information of scene,object and character are complementary to event image classification.Corresponding feature information of the event image is obtained from the scene,object and character feature extraction model respectively.The obtained feature information is fused by concatenation.The fused features are classified by softmax.Experiments show that the fusion of different features through stringing can effectively improve the classification accuracy of event images.Different visual information has semantic complementarity to the event image classification problem.3.Propose an event image classification method that fuses multi-CNN based on LSTM network.In order to effectively classify the event images,multiple image blocks are extracted from a given event image.Scene,object and character features are extracted from the image block using different convolutional neural network models.The features extracted from each image block are concatenated to obtain a representation of the corresponding image block.All image block representations of a given image are arranged in a sequence according to the position of the image block as input to the LSTM(Long Short-Term Memory),and the class of the corresponding event image is given by LSTM.Experimental results show that the above methods can effectively classify event images.