Encoding Word Order In Complex-valued Embeddings

Language model,especially the neural network language model,is crucial in Natural Language Processing and Information Retrieval.Recently Quantum Language Model(QLM)was proposed to unify single words and compound terms in the same probability space without having to extend the term space artificially as in previous studies.Neural Network based Quantum-like Language Model(NNQLM)extends QLM into an end-to-end architecture,building a density matrix through a bottom-up approach rather than complicated iterative estimation.Word position information is especially important for language models.However,QLM and NNQLM are only bag-of-words language model,and word position information is not considered.Currently,neural networks(NNs)address this by modeling word position using position embeddings.The problem is that position embeddings capture the position of individual words,but not the ordered relationship between individual word positions.We present a novel and principled solution for modeling both the absolute positions of words and their order relationships.Our solution generalizes word embeddings,previously defined as independent vectors,to continuous word functions over a variable(position).The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions.Hence,word representations in different positions can correlate with each other in a continuous function.The general solution of these functions is extended to complex-valued domain names complex-valued embedding,due to richer representations of complex number.We extend Convolutional Neural Network(CNN),Recurrent Neural Network(RNN),Transformer and NNQLM NNs to complex-valued versions to incorporate our complex-valued embedding.Experiments on text classification,machine translation,language modeling,Queation and Answer show gains over both classical word embeddings and position-enriched word embeddings.To our knowledge,this is the first work in NLP to link imaginary numbers in complex-valued representations to concrete meanings(i.e.,word order).