Research On Key Technologies Of Convolutional Neural Network-Based Short Text Classification

Text classification is an essential research subject in natural language processing,and it aims to classify the input text automatically into the categories.With the rapid development of the Internet,hundreds of millions of text messages are generated on the Internet all the time in the worldwide,and most of them are mainly short texts of no more than about 100 words,e.g.,social media blog posts,forum posts,web Q&A,product reviews.How to classify these massive short texts efficiently and accurately has become a very challenging task,which has attracted wide attention of researchers.In recent years,the research on key technologies of deep learning-based(especially convolutional neural networks)short text classification has made great progress and explosive breakthroughs,but the following problems still exist:(1)Traditional text representation methods are fragile to be affected by the size of data sets and length of input text,and they are prone to be dimension explosion,feature redundant or sparseness.Existing single-sense word vector does not consider the ambiguity of words;multi-sense word vector methods do not effectively make use of the influence of syntactic structure,word spacing and word order in the context on the semantic expression of words.(2)The context information provided by the short text is limited,so existing neural network models cannot extract abundant abstract features.How to build a more reasonable network structure,making convolutional neural networks more suitable for natural language processing is still an open problem to be solved.(3)Rectified Linear Unit(ReLU)has problems of “dying ReLU” and bias shift.Most newly improved activation functions abandon the sparse activation characteristic,but their performance on networks with different depth and structures are inconsistent.Other optimizations for the network structure tend to increase the amount of parameters and computational complexity of the network,making the model difficult to train.Therefore,to solve the problems above,this paper studies and explores several key technologies of short text classification based on convolutional neural networks.The main contributions are as follows:1.A multi-sense word vector calculation method based on gated convolution and hierarchical attention mechanism is proposed.The method is mainly based on a hierarchical attention gated convolutional neural network model.The calculation of the model considers the influence of contextual information such as ambiguous expression,word order,syntactic structure and word spacing on word meaning expression.The hierarchical attentional mechanism,composed of a sub-sense attention layer and a synthetic semantic attention layer,is constructed based on multiple gated convolutional layers encapsulated by non-residual block.Experimental results show that the multi-sense word vector calculated based on this method has a notable improvement compared with the baselines,and the hierarchical attention-gated convolutional neural network model has a significant improvement compared with other methods for predicting target words in language modeling tasks.2.A short text classification method based on attention-gated convolutional neural network is proposed.This method is inspired by the hierarchical attention gated convolutional neural network model.The current convolutional neural networks are difficult to down-sample the truly important abstract features due to the limited short text length.Based on the distributed hypothesis,and we introduce the attention-gated layer,to simulate the attention mechanism of human to control the influence of the target abstract feature,and help the pooling layer of the model to find the truly important features.Experimental results show that the attention mechanism of the attention-gated convolutional neural network is effective,and the method achieves significant improvements on standard convolutional neural network,and competitive results compared with other strong baseline models in multiple tasks.3.An activation function rectified by parametric natural logarithm transformation is proposed.This method introduces the parameter natural logarithm transformation to improve the x > 0 part while retaining the sparse activation characteristic.This method allows the activation function to be fine-tuned on different networks,shifts the mean activations of each hidden layer to near to 0,and reduces the variance,the bias shift effect and heteroscedasticity in the data distribution among layers,and alleviate the “dying ReLU” problem and gradient vanishing problem.Experimental results show that this method can improve the convergence performance of the network,accelerate the learning process and improve the performance of the network on multiple short text classification tasks.4.An optimization method for convolutional neural networks which we named N-fold Superposition is proposed.This method can reduce the noise in feature maps and improve the convergence performance of the convolutional neural network by feature map sharing and fully-connected layer weight sharing.This method does not significantly increase the amount of parameters in the network,and we prove that NS can make the model easier to converge and improve network performance by constructing more global minimum points of the loss function through the Fermat lemma and the extreme value judgment of the multivariate function.Experiments show that this method can reduce the noise in feature maps effectively,accelerate the convergence speed and improve the classification performance on multiple short text classification tasks of the convolutional neural network.