Research Of Text Classification Methods Combining Self-attention Mechanism And Convolution Optimization

With the explosive growth of text data scale,the application of text classification has become more and more extensive and important.Compared with traditional machine learning methods,the text classification methods based on deep learning reduce the dependence of complex feature engineering,and have significant classification effects.However,the text classification researches based on deep learning have the following problems: the text vectorization is usually realized by static coding,which is not well combined with specific contextual semantic information;the recurrent neural networks and convolutional neural networks have their own deficiencies;the spatial hierarchy of text is not fully utilized during model training process.This thesis focuses on the above problems,two models for single-label text classification and multi-label text classification are proposed,which combine the self-attention mechanism and convolution optimization.The main research contents are as follows:(1)Convolutional neural networks and recurrent neural networks have some defects,such as insufficient feature extraction and easy to cause the loss of important feature information.To solve above problems,this thesis proposes a multi-channel text classification model based on self-attention mechanism and Conditionally Parameterized Convolutions(Cond Conv).In this model,the character embeddings with rich semantic information are generated based on self-attention mechanism;the MGCond Conv method is proposed to generate multi-size convolution kernels with data dependence and extract local features of text with different granularity;the stacked Bi GRU with cross-layer connection is used to extract text contextual semantic information and long-distance dependencies,and the features of Bi GRU output are further filtered and purified by a max-pooling layer and a self-attention weight calculation layer.Experimental results on two Chinese datasets indicate that the proposed model is superior to the comparative models in terms of classification accuracy and F1-Score.(2)The multi-label texts have more features corresponding to different labels.When CNN performs pooling operation,it is easy to cause the loss of important text local features,while both LSTM and GRU have difficulty to model the hierarchical structure of text,so as to learn the deep semantic association information.To solve the above problems,this thesis proposes a multi-label text classification model based on the convolutional capsule network and ON-LSTM.On the one hand,the traditional convolution method is optimized,the max-pooling layer is replaced by the capsule network,which can extract and fuse the local features more comprehensively on the basis of preserving more feature information;on the other hand,ON-LSTM makes effective use of the text’s hierarchy to more fully capture the contextual semantic information and long-distance dependencies of the text.The experimental results on two multi-label English text datasets show that the classification effects of the model is better than the comparative methods or models.In addition,through experiments,it is found that when there is a strong correlation between class labels,the introduction of label correlation calculation is helpful to improve the classification effects.