Research On Neural Sequence Prediction Model

With the development of deep learning and natural language processing techniques,the neural sequence prediction model has become a research hotspot and has been widely used,such as machine translation,auto-summarization,and image captioning.In recent years,the sequence prediction model has not been limited to natural language processing tasks,and has gradually been introduced into the recommendation tasks.In this paper,we summarize the existing neural sequence prediction model,and make improvements to several problems in text sequence prediction and behavior sequence prediction.The work of this article can be summarized as the following two parts:1.We propose the exposure bias regularization framework for the training of the sequence prediction model.We point out the inconsistency of the input distribution when training and testing the sequence prediction model,and propose an exposure bias regularization term during training to improve the generation ability of the model during inference.Besides,in order to alleviate the problem of slow convergence,we also propose two auxiliary training methods parasitic learning and curriculum learning to help the model training under the exposure bias regularization framework.2.We propose the Time-sliding Window based mini-batch Training and Time-aware Long Short-Term Memory Network to improve the efficiency and effectiveness of behavior sequence prediction model.Different from text sequence,behavior sequence has larger variances in length and uneven time interval.We modify the sequence prediction model to make it adapt to the characteristics of the behavior sequence.There are two main improvements:First,for the problem that the length of user behavior sequence can be very different,we propose the Time-sliding Window based mini-batch Training to improve the efficiency of the model training.Secondly,for the problem that the time interval in the behavior sequence is irregular,we propose Time-aware Long Short-Term Memory network to use the time interval to balance the user's long-term and short-term interests.Finally,we apply the sequence prediction model to the user behavior sequence modeling,which is served as a recall model in the recommender system.