Research On Contextualized Word Representations And Domain Transfer

With the rapid development of social informatization,a large amount of text data containing rich knowledge and information is generated every day.The use of computer and natural language processing for automatic language analysis and text processing promotes human life rapidly.However,to perform natural language processing,the basic components of the language must be transformed into mathematical representations that the computer can recognize.The contextualized word representation method solves the problem of "word polysemy" that can not be solved by the traditional pre-training static word vector representation.It can describe the change of a word meaning in different contexts and its complex grammar and semantic information.Different from the traditional pretraining word vector method,the pre-training output of the contextualized word representation is a sentence semantic representation model,and the deep-semantic fusion of each word of the sample sentence is performed using the pre-training model parameters when applied to the downstream task.The performance of various tasks such as syntactic analysis,semantic role labeling,automatic question and answer,textual entailment,and reading comprehension have been significantly improved.Chinese word segmentation is an important basic task of Chinese information processing.Previous work is basically based on word vector representation modeling,and it is difficult to overcome many of the above drawbacks.Starting from the Chinese word segmentation problem,this thesis constructs the word segmentation model by using the pre-training contextualized word representations and conducts experiments in multiple Chinese word segmentation datasets.Our model has been significantly improved compared with the traditional model in word segmentation performance,by taking advantage of the strong generalization ability that the pretraining contextualized word representation brings.The problem of domain transfer is also the focus of our research.Due to the ambiguity of the natural language domain,the model trained from general corpus shows a serious decline in performance in the professional domain.Professional domain data is often difficult to obtain.If you can transfer general domain knowledge or models to professional domain better,you can solve the above problems.We conducted a domain transfer experiment on the word segmentation model using contextualized word representation and compared it with the traditional word segmentation model in different transfer dataset scales.The results show that the word segmentation model based on deep contextualized word representation is far better to the traditional models in terms of cross-domain generalization ability and transferability,and can perform more efficient stable transfer with fewer data.This kind of depth contextualized word representation model is based on unlabeled corpora,and it is difficult to combine other effective features.Simply concatenation of hidden layers usually does not bring good results.Moreover,due to the complexity of its model,there are problems such as excessive memory and computational resources,and slow prediction speed.In this thesis,we use the multitask learning framework to solve these problems.On the one hand,we use multi-task learning to add feature knowledge to the contextualized word representation model,which makes the feature knowledge transfer and merge with other tasks,and achieves better performance than the model that trains independently and concatenates simple features;on the other hand,the multi-task model greatly speeds up the efficiency of multi-task text analysis,since it shares representation and calculation for hidden layers.