| Image caption is a comprehensive problem integrating computer vision,natural language processing and machine learning.Its purpose is to translate the content of the image into a natural sentence description that conforms to human habits.It can be regarded as a task to describe the image information.It can automatically generate a description for the visual information.In recent years,the main trend of extracting word features from pictures by convolution neural network.At present,most image caption methods are based on two-stage encoding and decoding model.However,there is still a certain gap between the current mainstream methods for image feature processing and the effective extraction of semantic information in images,so the output results are not idealFirstly,this thesis studies the image description methods in recent years,it is found that the current mainstream models mainly extract a single feature information in the image feature extraction stage.The extracted features are too single in type and scale to generate a comprehensive image description.And a single image feature often leads to recognition errors and text output errors.For images with rich content,single type of feature cannot cover the complete semantic information of the image,and there are some defects.This thesis considers that after extracting the two features respectively,the self-attention module is used to allocate the image feature weight to fuse all the features,to obtain as rich and accurate semantic information as possible.Secondly,this thesis studies how to enhance the detailed description of the object in the generated statement,make the object description more detailed,and make the output description more complete in the local object description.At present,the features used in image caption direction are image features extracted by convolution neural network and regional proposal network.Then use it to generation natural language description.To make the model be able to filter the primary and secondary information in the visual information and eliminate the noise information and redundant information by controlling the same as people.However,the current attention mechanism takes the useful information and redundant information in image features as input features at the same time,and the existence of redundant information directly affects the accuracy of the model,resulting in the neglect of some semantic information.This thesis improves the correlation between results and features from the perspective of location clustering,so that the model can learn the complementary relationship between object regional features and grid features containing their detailed information,to improve the integrity of model output.Finally,this thesis trains on Image Net dataset,visual Gerome dataset and MS coco dataset,and tests of Image caption on MS coco dataset.The effectiveness of this model is verified by comparing with previous research methods. |
