Language Modeling By Tensor Space And Tensor Network

Language modeling(LM)is a fundamental research topic in the field of natural language processing.It is applied in many natural language processing tasks,such as speech recognition,machine translation,dialogue system and so on.The existing language models can be divided into statistical language models and neural language models,each of which has its own advantages and disadvantages.The parameters required for statistical language models are too large to be estimated.Although the neural language model has good modeling effect,it is insufficient in theory.In this work,we propose a language model named Tensor Space Language Model(TSLM),by utilizing tensor networks and tensor decomposition.In TSLM,we capture the interaction information between words through tensor product,based on which we can build a high-dimensional semantic space with stronger expressive power.Theoretically,we prove that such tensor representation is a generalization of the n-gram language model.We further show that this high-order tensor representation can be decomposed to a recursive calculation of conditional probability for language modeling.In summary,our proposed tensor space language model is a more general language model,which means that both statistical language model and neural language model can be used as special cases of tensor space language model.In other words,the tensor space language model we constructed unifies the statistical language model and the neural language model.The experimental results on Penn Tree Bank(PTB)dataset and WikiText benchmark demonstrate the effectiveness of TSLM.