Sentence-Level Language Analysis With Contextualized Word Embeddings

Natural language processing(NLP)is an important sub-field of artificial intelligence.As the first step of automatic text processing,converting a word into its computerunderstandable representation greatly affects the quality of NLP.The word vector gives the smallest semantic unit of natural language — the word a dense vector representation which contains syntactic and semantic information.As the basic block of neural-based NLP systems,the word vector which is computed according to the distributional hypothesis of words,has brought performance improvement to many natural language processing models.To improve the training efficiency,previous work to make static assumptions on word vectors,that is,a word has a unique vector.This assumption makes it possible to learn word embeddings on large-scale data,but it also makes it impossible for static word vectors to determine their representations on context,and thus cannot model the “polysemous words”.The contextualized word embedding,as an emerging technique,cancels the static word vector hypothesis and dynamically determines the vector representation of a word on its context.Contextualized word embedding has helped improved a range of tasks,including questions and answers,textual implications,and sentiment analysis.Many NLP tasks rely on sentence-level language analysis including Chinese word segmentation,part-of-speech tagging,named entity recognition,and syntactic parsing.Improving the performance of language analysis help improve the performance of natural language processing.In recent years,the help of static word embeddings help neural language analysis algorithms to achieve performance improvement.However,the role of contextualized word embeddings for language analysis is yet to be explored.Based on the progress of contextualized word embeddings and language analysis technology,this paper focus on the combination of these two techniques.The research of this paper mainly includes the following directions:1.Localized and contextualized word embeddings: Contextualized word embeddings suffer from efficiency problem,which is mainly caused by the practice of modeling the global context with multiple layers.Considering that the sequence labeling problem largely depends on the local context,we combine the relative position and the local self-attention to represent the context in the contextualized word embeddings.Experimental results on five sequence labeling problems show that,thanks to the local contextrepresentation,our model runs three times faster over ELMo without loss of accuracy.2.Lexical analysis with contextualized word embeddings: Properly representing a segment(e.g.word for Chinese word segmentation and entity for named entity recognition)is important to the segmentation performance.We propose to represent a segment by simply concatenating the input vectors and apply this representation to neural semi-CRF.Experimental results on a collection of segmentation problems show the effectiveness of our method.By incorporating additional segment embedding which encodes the entire segment,our model achieves comparable performance the state-of-the-art systems.3.Syntactic analysis with contextualized word embeddings: The effect of contextualized word embeddings on universal parsing is yet to know.We propose to use the contextualized word embeddings as an additional features and show its effectiveness with experiments on extensive treebanks.We further explore the reason of performance improvement.The analysis shows the improvements are resulted from the better out-of-vocabulary word abstraction from the contextualized word embeddings and such abstraction is achieved by better morphological modeling.4.Distilling knowledge from syntactic parser with contextualized word embeddings: Over-parameterization slows down both the training and testing of the model with contextualized word embeddings.We proposed a knowledge distillation method for this kind of search-based structured prediction model,which distills a complex predictor into a simple one,thus speeds up the model.Our method combines the distillation from reference states and explored state,which shows to improve the performance.Experimental results show that our method achieves magnitude of speedup with a slight loss of accuracy.In general,this paper studies contextualized word embeddings and their applications in language analysis.contextualized word embeddings significantly improve the performance of linguistic analysis,while linguistic analysis provides a new understanding of contextualized word embeddings.This paper can make language analysis techniques better serve the downstream tasks of natural language processing,thus promoting the development of the entire field.