The Research Of Applying Conditional Random Fields To Chinese Lexical Analysis And Chunk Parsing

This dissertation introduces the research of lexical analysis and syntax parsing is important, crucial and fundamental in the research of natural language understanding. According presently the tendency of methods that integrate statistics-based and rule-base methods, this paper introduces the rules of Maximum Entropy and the significance of it on natural language understanding research. Furthermore, this dissertation discusses the definition and parameter estimate of Condition Random Fields. CRFs are probabilistic models for segmenting and labeling sequence data and heavily motivated by the principle of maximum entropy. Then this dissertation presents a unified approach for Chinese lexical analysis using Conditional Random Fields. Precious applications applying conditional random fields to Chinese words segmentation convert segmentation to character-based Begin/Inside tagging. This dissertation presents using the words lattice as the fundamental sequence to be tagged to achieve Chinese lexical analysis. Then the lexicon can be used efficiently, and language knowledge can be integrated easily in feature template selecting. This dissertation also discusses applying Conditional Random Fields to Chinese Chunk Parsing and our future works.