Research Of Image Abstract Generation Based On Deep Learning

Vision is the main sensory modality of human perception,and language is the most powerful tool for human communication with the world.Image abstraction is a combination of machine vision and natural language processing.Simply put,a computer is fed an image that produces a fluent natural language description of the content.Image abstract generation technology has achieved good results in the fields of search engine,blind assistant hearing and intelligent robot,and has a very promising application prospect.The traditional image abstract generation technology mainly includes two methods based on template and retrieval,but both of them have obvious defects.In recent years,deep learn-based methods have made remarkable achievements in image abstract generation technology.In this paper,a new image abstract generation method is proposed,which improves the model based on the existing model and aims at the main shortcomings.The work is as follows:In this paper,based on the classical framework of encoder,decoder improved a method to generate the image,the image encoder part,using the Convolution Neural Network(CNN),and joined the properties prediction layer,so that you can convert image feature to high-level semantic expression,then input to the decoder,so as to improve the performance model.Decoder part in language,the first to use the double Long Short-Term Memory(LSTM),solved the shortcoming of single-layer network poor expression ability,and then introduces adaptive mechanism,attention is decoder in the generated description statement can adaptively choose using image information,makes the result was a more accurate statement.Simulation results show that the improved model has a higher evaluation score in MSCOCO dataset than other mainstream image abstract generation methods.This paper introduces reinforcement learning to optimize the improved model.Although the evaluation score of the improved model is higher than that of other mainstream image summary generation methods,there are still some problems: the first is "Exposure bias" problem,and the second is the target function used in training and the index mismatching problem in test.In order to solve the above two problems,this paper USES reinforcement learning algorithm,takes the improved model as the pre-training model,and then directly optimizes the CIDEr evaluation index to re-train the model with the strategy gradient algorithm,and obtains the final model.Simulation results show that the performance of the model is improved by reinforcement learning.Finally,an english-chinese translation based on Seq2 Seq model is added.Because of the difference between Chinese and English in grammar,it is more difficult to generate image abstracts in Chinese,and most of the training data sets are in English.If an individual makes a Chinese data set,the workload will be very large.To solve this problem,this paper constructs an english-chinese translation model based on Seq2 Seq model,and takes the English abstract of the image generated in the previous part as input to obtain the translated Chinese,thus realizing the Chinese abstract of the image.The simulation results show that the English-Chinese translation model is effective in the test set.