Improved Recurrent Neural Networks And Its Application In Chinese Language Processing

How to represent the different language units of Chinese and how to model the relationships among these units are the basic tasks of Chinese language processing.Recurrent neural networks(RNNs),as a class of Turing complete machine learning model,can provide a framework that can learn the representation of language units and simultaneously model the relationships among these language units.However,current researches on RNN and RNN-based Chinese language processing are insufficient because: 1)the memories of existing RNNs are limited;2)existing neural language models for Chinese representation learning rely only on text corpora,which will make it hard to reflect the semantics of Chinese linguistic knowledge,and the direct prediction of target words by neural language models will cause the dimension of softmax function of the output layer to be too high;3)existing RNNs have insufficient ability to model long-term dependences(LTDs)in Chinese.Correspondingly,this dissertation focus on the following aspects:· A general definition of memory-augmented recurrent neural networks(M-RNNs)is proposed,and three analytical indicators named duration,addressability,and capacity of general forms of the additional memory in M-RNNs are formalized.The three indicators are used to measure three aspects of M-RNNs: the duration of memory content,the complexity of memory accessing and the available information of memory over a period of time.Based on these indicators,the DAC(Duration,Addressability and Capacity)principle is discovered,which reveals that it is hard for an M-RNN to simultaneously provide good performance on more than two out of three of indicators.· To improve the Chinese representation learning,two approaches of using the morphological and phonological knowledge of Chinese characters are proposed,i.e.,knowledge as additional supervised signals and knowledge as an external memory.Two improved RNNs: Mor Pho RNN(Morphology and Phonology enhanced RNN)and MorPho M3-RNN(Morphology and Phonology Multimodal Memory-augmented RNN)are designed respectively.The results of experiments show that the morphological knowledge and phonological knowledge of Chinese characters has a significant improvement in the performance of Chinese representation learning.In addition,by predicting the subcomponents and the Pinyin information of the characters of a target Chinese word instead of predicting the target word directly,it is achieved two advantages: 1)the dimension of the softmax function of the output layer of the model is reduced;2)the dimension of the output layer of the model is fixed and does not increase with the increase of the word list.· An improved M-RNN called LWM-RNN(Long-term and Working Memory-augmented RNN)is proposed for Chinese sequential labeling tasks.Considering the possible LTDs in Chinese sequential labeling tasks,on the one hand,the new architecture can separate information processing and information storage;on the other hand,the information storage is divided into long-term memory and working memory according to the duration of information.Additionally,it is proved that LWM-RNN can learn LTDs without suffering from vanishing gradients with necessary assumptions.The experimental results on the two Chinese sequential labeling tasks(Chinese word segmentation and named entity recognition)show that LWM-RNN has a potential advantage for long sentence processing and text discourse-level Chinese information processing tasks.