Research On Attention-based Model For Sequence Classification

Sequence classification is an active research in the field of data mining.The essence of this task is to assign a predefined label to a given sequence.There are many researches in the field of sequence classification,which can be broadly divided into the following three categories: feature-based,distance-based and model-based methods.The first two methods are time-consuming and require professional background knowledge due to the design of handcrafted features and distance calculation.At the same time,these methods are difficult to model the dependencies of sequences and extract multi-scale features.With the development of deep learning,neural networks can extract the sample features end-to-end through its model structure and back-propagation algorithm.In addition,the sequence data contains abundant information,but most of the current classification methods cannot adaptively mine the sequence data for specific samples.However,the attention mechanism,as a method to imitate biological observation behavior,has developed rapidly in the field of computer vision in recent years,which can adaptively select the appropriate features for specific sample.Therefore,this paper will explore the direction of deep learning combined with attention mechanism.The main research work includes the following two aspects:(1)An Attention-based Variable Correlation Mining Network(VCM-Net)is proposed to solve the multivariate time series classification tasks in this work.The method first extracts the variable correlation features through sparse multi-scale convolutional filters,and then uses the attention mechanism to select these features for each sample.In addition to the adaptive mining of variable correlation features,VCM-Net also models the dependence of time series.And the experimental results on 18 benchmark multivariate time series and 3 skeleton sequences demonstrate that the proposed method has good performance.Furthermore,experimental analysis and visualized analysis verify the effectiveness of extracting and selecting the correlation features of variables.(2)An Adaptive Contextual Representation Learning Network(ACRL-Net)is proposed to solve the text classification tasks in this work.The model is composed of two components,a context encoder and an adaptive decoder.First,the data flows are designed in the encoder to construct two different contextual representations.The attention mechanism is then used to select appropriate contextual representation features for each sample.The experiments are conducted on 3 large-scale,4 small-scale and 16 Amazon Product Reviews datasets,and ACRL-Net achieves good performances on most of them with different properties,demonstrating its strong adaptability.At the same time,the effectiveness of adaptively selecting contextual representation for text classification is verified by experimental analysis and attention weight visualization.