A Method And System For Generating Poetry From Images Based On Deep Learning

In recent years,research on the automatic generation of classical Chinese poetry based on deep learning has gradually become a hot topic.Most of these studies focus on the automatic generation of poetry from textual information.In this paper,we focus on generating poetry from image information,which is a cross-modal task.At present,research works on this task still have the problems of topic shifting and semantic inconsistency;while some important image information cannot be accurately represented in the generated poems.Moreover,the paired datasets of images and poems have poor quality and are difficult to construct during the training process.To tackle these problems,we propose a method for generating poetry from images based on deep learning,first we define image information as concrete and abstract information,and then extracting and integrating the two kinds of information from images.We set up several groups of comparison experiments and ablation experiments to evaluate the performance of the model through machine evaluation and human evaluation.The evaluation results show that our method outperforms other methods,and verify that the method of this paper could effectively improve the consistency of images and poems without losing the quality of the generated poems.The main contributions of this paper are as follows:1.We propose a fill-in-the-blank poetry generation model.The model fills the keywords of concrete information into each line of poetry in an explicit way,which ensures that the keywords extracted from the images must appear in the generated poems.In this way,we solve the problem of topic shifting.2.We propose an abstract information integration method based on word embedding,which integrates abstract information into the generated poems by means of word embedding.In this way,we solve the problem that the abstract information in the images cannot be accurately represented in the poems.3.We use a special training method to align image and poetry information,using nonparallel data in the training process instead of paired datasets of images and poems;special image datasets and poetry datasets are constructed separately in order to train the model.This paper demonstrates a Chinese classical poetry generation system based on deep learning.Existing systems for Chinese classical poetry generation are mostly template-based and very few of them can accept multi-modal input,these systems also have low degrees of flexibility and cannot achieve collaborative human-computer poetry writing.In order to improve the shortcomings of existing systems,our system uses neural networks that are trained on over200 thousand poems to achieve better generation results.Our system can accept multiple modal inputs,such as text,image,or artistic conceptions.The system also adds the function of assisted poetry composition,where the user can collaborate with the system to create a poem and the system acts as an assistant tool to help the user to complete the composition.For the user’s convenience,we deploy the system at the We Chat applet platform,users can use the system on the mobile device whenever and wherever possible.