Research On Chinese Word Segmentation Methods Based On Deep Learning

As text data continues to increase,there is an increasing need to use natural language processing techniques to process textual information.Most Chinese natural language processing tasks must be segmented before they can be processed in the next step.The result of segmentation has a great influence on the subsequent work,so Chinese word segmentation becomes the basic task of natural language processing.Chinese word segmentation technology continues to advance,from dictionary rule methods to statistical learning methods,to today's deep learning methods,the problems in word segmentation are constantly being solved.The deep learning method is the main method of Chinese word segmentation nowadays,which solves the problem of feature engineering in statistical learning methods.The main structure is a Bi-directional long and short memory network.This network structure can obtain text information with time series and achieves good results for Chinese word segmentation tasks.There are also many problems with the use of Bi-directional long and short memory networks for Chinese word segmentation.The first is the data problem.If the data is not enough,it is difficult for the neural network to train good network parameters.The second is the speed problem.The Bi-directional long and short memory network is a sequence model,and the training speed is very slow.In this paper,some research work has been done on the above two problems of Bi-directional long and short memory networks.Regarding data problems,the common practice is to introduce dictionary data,so this paper also uses external radical information,and uses the Bi-directional long and short memory network to deeply extract the radical information,and then fuse the radical information and character information into joint information.This neural network is used to extract features to enrich the data features,and then the joint information is sequenced as the input features of the BiLSTM+CRF model.Regarding the speed problem,the slow speed of the Bi-directional long and short memory network is determined from the beginning of the design of the recurrent neural network.It is difficult to greatly improve the speed by optimizing the structure of the recurrent neural network.This paper uses the dilated convolutional neural network to replace the Bi-directional long and short memory network.The neural network has a natural advantage in training speed,and the design of the dilated convolutional neural network can obtain text information with a longer length.In this paper,experiments were carried out on four data sets of PKU,MSR,CITYU and AS.Multiple contrast experiments were designed to verify the effect of adding radical information,adding CRF effects,and using the dilated convolutional neural network.Experiments show that adding radical information and CRF,using dilated convolutional neural network,can improve the word segmentation effect.and the precision rate,recall rate and F value of the four data sets are improved.The main advantage of the dilated convolutional neural network is that the speed is about twice as fast as Bi-directional long and short memory network.The paper has 31 pictures,22 tables and 52 references.