Text Classification Research Based On Hierarchical Transformer With Contrastive Loss

As the society enters the era of information explosion,a large amount of information has emerged on the Internet,most of which is presented in the form of text.Natural language processing,as an effective means of text analysis,has therefore developed rapidly.As a classic natural language processing task,text classification technology is relatively mature,but there are still some shortcomings:1)The traditional text classification paradigm rarely utilize the correlations between samples and labels,with label information only considered in the final classification loss calculation;2)Supervised text classification relies on annotated data with high cost and limited scale,while unannotated data and increasing computing resources cannot be fully utilized;3)One-hot encoding of labels in text classification has defects in the description of real-life categories of text,making it difficult to adapt to all application scenarios.To address the aforementioned shortcomings,this paper proposes a text classification method based on Hierarchical Transformer model with Contrastive loss(HTC),using a pretrained language model as the underlying encoder to obtain initial vector representation of a single text sample.The main contributions are as follows:1)A hierarchical Transformer structure is proposed to model the relationship between samples and samples,as well as samples and categories,to improve classification performance.The self-attention mechanism is utilized to adaptively integrate rich information including category labels and unlabeled text samples to assist in training sample encoding.2)Contrast learning is introduced to construct self-supervised signals to mine similarity information between samples.At the same time,Memory Bank is adopted to completely decouple Batch size,the number of samples used by upper-layer Transformer,and the number of negative samples used in the contrastive loss,effectively utilizing unlabeled data for enhanced training under limited resource conditions.3)For the broad text classification tasks such as textual personality and emotion analysis,an HTC-LSTM model that explicitly considers the correlation between categories,is proposed for comparison with the HTC model,to demonstrate the adaptability of the HTC model in complex scenarios involving multiple categories and interrelated category information.Finally,experiments were carried out in complex and diverse application tasks such as news classification,medical record classification,personality recognition and emotional distribution prediction.The experimental results show that,through comparison with similar models and ablation experiment,the proposed model achieves improvements in multiple performance metrics,which proves the effectiveness of the model.