Text Classification Based On Label Embedding And Attention Mechanism

With the development of internet,large quantities of texts are generated all the time in the world.In these chaotic document resources,there is actually a wealth of commercial value,but the premise is that they need to be effectively organized and managed.In the face of massive texts,manual processing is obviously not advisable,and automatic text classification came into being and has become an essential tool for the management of text information.There are endless technical researches on text classification.In traditional test classification,text representation methods such as one-hot encoding usually suffer from data sparseness and dimensionality problems,while feature extraction algorithms such as convolutional neural networks cannot focus on important features.Recently,the pre-training model and attention mechanism have been fully exerted in the field of Natural Language Processing and have achieved great success.As a result,they have gained more extensive attention and research.Based on two classic models,convolutional neural network and recurrent neural network,this dissertation proposes two text classification algorithms combining pre-training models such as BERT(Bidirectional Encoder Representations from Transformers),label embedding and attention mechanism,which as follows:(1)Label-Embedding-Based-Multi-scale Convolution for Text Classification(LEMC).The LEMC model uses the K-means algorithm to cluster all samples in the data set while capturing the local semantic relationship through the convolution operation,and uses the obtained bi-gram vector to initialize the convolution kernel so that the model can focus on learning important semantic features at the beginning of training.The embedding of label information improves the traditional text representation,which can better reflect the category attributes of the text itself.Experimental results on five classification tasks show that the LEMC model is superior to classic text classification models such as convolution neural networks,indicating the effectiveness of convolution initialization and label embedding.In addition,neither convolution initialization nor label embedding uses external resources,and only requires a relatively small amount of calculation,which is very attractive for situations where training costs may be problematic.(2)Text Classification under the Attention Mechanism Based on Label Embedding(ALEC).The ALEC model can effectively capture long-distance semantic features and sort and learn the importance of sequence output features through the attention mechanism.The algorithm uses the BERT model to perform word vectorized representations of text and labels at the same time,and multiplies the obtained text matrix and label matrix to obtain an interaction matrix,and then introduces the idea of attention mechanism,and uses convolutional neural network to extract the features from the interaction matrix and obtain the correlation score between the text and the label to improve the sequence output feature of the bidirectional long short-term memory network with the text matrix as the input.Experimental results indicate that the ALEC model outperforms other advanced models,and the test accuracy has been greatly improved.