An Approach To Compress The Low Frequency Words In Neural Conversation System

Recently,generation-based neural conversation systems have attracted more and more attention from both academia and industry.A neural conversation system could be modeled as a sequence-to-sequence task.Hence,an input message from the user is regarded as a source sequence,and a generated response is regarded as a target sequence.Generally,the most common way to construct a generation-based neural conversation system is implementing with the Encoder-Decoder framework.In this framework,the Encoder summarizes an input sequence into contexts,then the Decoder utilizes these contexts to generate a new word sequence as a response.Previous work tends to operate an Encoder-Decoder model at word-level,which means both of the Encoder and Decoder represent an input/output as a sequence of words.Due the computational complexity,both of the Encoder and Decoder operate words by maintaining a vocabulary of a fixed size,which is much less than the total account of words of a language.Obviously,out-of-vocabulary words(OOVs)will become unknown words in an EncoderDecoder model.However,since a model could not obtain the corresponding vectors of OOVs by looking up vocabulary,Encoder could not recognize OOVs and Decoder could not generate OOVs,which seriously impacts the generation quality of a conversation system.This paper proposes a low frequency words compression based hybrid-level model HLEncDec to address this issue and improve the generation quality.In HL-EncDec,OOVs are classified as low frequency words,which will be compressed to character-level based representations.In the Encoder side,a low frequency word could be recognized by utilizing a convolutional neural network based approach and its characters to calculate an equivalent word vector.In the Decoder size,to generate a low frequency word,it will be decomposed as several characters.Subsequently,this paper also proposes an improved version of HLEncDec+,which fuses the way to obtain a word vector for both high frequency words and low frequency words.We conduct several experiments to evaluate HL-EncDec and HL-EncDec+ on a Chinese corpus that consists of more than 3M conversational pairs.Experimental results of both automatic metrics and human annotations show our models significantly outperform baseline models.