Improving Word Vector Model With Part-of-Speech And Dependency Grammar Information

Recently,with the help of distributed word vector based on neural network,DeepLearning has made great progress in the field of natural language processing,sweeping over the basic research of natural language processing.The quality of word vector directly affects the effect of upper natural language processing tasks.As the key of word vector,word vector models suffer from following challenges:(i)Network structure problems.Nowadays,most of the word vector models based on neural network treat words in context windows equally,without considering the dependency relations between words.(ii)Information loss problems.The fixed context windows will prune long sentences and compound sentences,which contain complex sentence components,resulting in the loss of some important words.(iii)The lexical structure information of Part-of-Speech tagging and the syntactic structure information of dependency parsing are not fully utilized.Most of the existing researches using Part-ofSpeech tagging to advance the word vector models only use Part-of-Speech association to modify the weight of words and most of the existing researches using dependency parsing to advance word vector models does not consider the differences of dependency relations.(iv)The technology of sub-sampling and negative sampling is too simple.Words with high word frequency are treated equally when they are used sub-sampling and nagative sampling,which results in the loss of some high-frequency words that have great influence on predicting the target word.v)The similarity between Part-of-Speech cannot be measured.There is a semantic gap between Part-of-Speech,as far as we know,there is no algorithm or dataset to quantify the similarity between them.This paper proposes four improved word vector models on the basis of existing work,aiming at the problems faced by the word vector models based on neural network,combining Part-of-Speech tagging and dependency parsing.They are:(i)CBOW+P model based on Partof-Speech tagging.Part-of-Speech information is integrated into the training process of word vectors,and the concept of Part-of-Speech vectors are proposed to solve the problem that the similarity of Part-of-Speech is difficult to measure.The Part-of-Speech vectors correlation coefficient and distance weighting function are used to train the Part-of-Speech vectors as well as the word vectors in a unified way.Meanwhile,the Part-of-Speech ratio is used to improve the sub-sampling and negative sampling techniques.(ii)CBOW+PW model based on Part-of-Speech tagging,which is based on CBOW+P model,where the correlation coefficient of Part-of-Speech vectors are further refined into each word.(iii)CBOW+G model based on dependency parsing.In this paper,dependency parsing is used to correct the information loss caused by fixed context windows.Dependency relations weight is used to measure the difference of dependency relations.At the same time,two strategies for calculating dependency relations weight are proposed: Pre-Training Cosine Distance strategy and Negative Sampling Cosine Distance strategy.(iv)CBOW+G+P model based on Part-ofSpeech tagging and dependency parsing,which fuse the CBOW+P model and CBOW+G model.Both Part-of-Speech tagging information and dependency parsing information are used to improve the word vector model.In order to measure the effect of Part-of-Speech vectors,this paper constructs a Part-ofSpeech analogy dataset with 55 groups test data and a sentence representation method based on Part-of-Speech vectors.Experiments on word similarity,word analogy and Chinese text categorization tasks verify the validity of the proposed models,especially the superiority of combination model(CBOW+G+P),which has the same magnitude of time complexity as CBOW model.