Research On Chinese Short Text Classification Based On Pre-trained Language Model

The text classification method based on deep learning has been the focus of the current text classification field.this method is to convert natural text into digital text,and then extract text features,and build the corresponding neural network to train the model to achieve better text Classification effect.the current best deep learning model is a pre-trained language model(BERT,ALBERT,etc.),this model can reduce the training time of text classification and improve the training effect of text classification tasks.but there is also an effect of processing Chinese short texts Poor,poor processing of complex emotional words and many other issues(the time sequence information of the text is not enough when processing text.When performing multi-label emotional classification,when the number of tags is large,each classification index is not high),for improvement For the above problems,the thesis has done some work in the following aspects:(1)When the pre-training language model BERT performs single-label text classification training,it occupies too much hardware resources and does not make good use of the timing information of the text.To solve this problem,a text classification model based on ALBERT’s word vector and sentence vector combined with Bi GRU(ALBERTcls+Bi GRU)is proposed.Compared with BERT,this model takes up less hardwar0 e resources during training,and makes full use of the timing information between words.Compared with the pure CNN,RNN,RCNN,BERT and ALBERT models,the model has a good classification effect in the classification of Chinese single-label short text.(2)Because the structure of the pre-training language model is a lot of Transformer structures overlapping,some sentences have similar meanings,and the sentence vectors converted are similar.However,these vectors may be transformed into vectors with larger differences after a complex overlapping Transformer structure,which in turn affects the final classification result.In response to this problem,this paper proposes a pre-training language model combined with aggregation operations(such as Maxpooling,Meanpooling)model to correct the sentence vector of the pre-training language model,and achieve better than the traditional RNN,CNN,RCNN and pre-training The language model performs more excellent results for complex operations.This model not only improves the effect of the model,but also minimizes the complexity of the model.(3)For multi-label sentiment classification,when the number of labels is large,each index is not high.A method based on pre-training language model combined with Stacking ensemble learning is proposed to integrate multiple high-efficiency models based on pre-training language models.Complementary advantages,correct each other,and improve the classification effect.It was verified by experiments that these models performed best on a certain index or on a certain label,allowing them to correct each other,and finally the experiment achieved excellent results.