Research On Relationship Energy Maximization Based Word Embedding

Word embedding refers to the technique of expressing words as low-dimensional vectors and reflecting the relationship between words in vector space.Word embedding technology can be applied not only in the field of natural language processing,but also in other fields such as recommendation and advertising.Among the many tasks in these fields,the word embedding technique is generally used to pre-train the word vectors and use them as input to the downstream model.Traditional word embedding algorithms,whether based on co-occurrence matrix or neural networks,are lacking in interpretability.Secondly,since word embedding can be applied to the fields such as recommendation and advertising,the industry requires that word embedding technology can learn more word relationship information to promote the effect of these tasks and bring more benefits to the enterprise.The theoretical basis of word embedding is the word distribution hypothesis proposed by Harris et al.,that is,"words with similar contexts are more similar in semantics".This hypothesis reflects that the relationship information between words is represented by the similarity of its context words.If we trace the source,based on the word distribution hypothesis,in-depth study of the relationship between words and their context words may be an effective way to improve the interpretability and learning ability of word embedding technology.Inspired by this,this thesis starts from the word distribution hypothesis and carries out related research on the relationship between words and context.The main work is:(1)According to the interpretability of the word embedding method,combined with the word distribution hypothesis theory and the traditional word embedding method to obtain the word vectors,the relationship energy maximization based word embedding method is proposed.First of all,according to the theory that "words with similar contexts are more similar in semantics",all the word relationships of the entire corpus are represented by their contexts,and the energy formula of the relationship is established.Secondly,using the maximum likelihood estimation,the gradient updates the corresponding word vector for each word.Finally,the formula is transformed to transform the process of maximum likelihood estimation into a process of constructing an energy matrix and performing matrix decomposition.The algorithm proposed in this thesis reflects the theory of word distribution hypothesis in the process of acquiring wordvector,which is more explanatory.Using the deep learning common dataset Text8 as a corpus,the algorithm of this thesis is better than the traditional matrix decomposition algorithm on the Word analogy task,and the Word similarity task can be equivalent to the Word2 vec task.(2)In view of the application effect of word embedding technology in the field of recommendation and advertising,this thesis applies the word embedding method based on relationship energy maximization to the field of recommendation and advertising,and uses the search advertisement click and conversion data in Alibaba algorithm competition to conduct experiments.First,establish a conversion rate estimation model and build rich features from the data to make the model's prediction effect reach a better level.Next,the items and attributes are constructed as words and contexts and the matrix is decomposed to obtain a low-dimensional vector representation of the items.Finally,the vectors of the items is added as a feature to the conversion rate prediction model to measure the effect of the model.Through experiments,this method can further reduce the Logloss of the conversion rate prediction model,and achieve better results than the traditional word embedding technology.