Improvement And Compression Of Pre-Trained Language Models For User-Generated Texts

Document-level sentiment classification,an important task in the field of natural language processing,has always been widely concerned.Thanks to the development of Internet technology,the texts published online by users continue to accumulate and have provided a solid data foundation for related research.With the improvement of computing power and the introduction of the Transformer structure which is stackable,a large number of Pre-trained Language Models(PLMs)began to emerge over the past few years.These models usually have deep layers with a lot of parameters and have been pre-trained on a large-scale corpus.Therefore,they achieve great performance on a variety of NLP tasks since they were proposed.In recent years,study on the enhancement of PLMs for better performance and the compression of PLMs for better efficiency has gradually become research hot spots.However,existing works rarely make exploration in the following two aspects:(1)How to enhance PLMs using user information of the text.(2)How to compress PLMs considering the privacy of user-generated texts.Therefore,in the task of sentiment classification for user-generated texts,this paper studies the two problems mentioned above based on the PLMs.The main work and contributions of this paper are summarized as follows:The first work is a study on the optimization of PLMs based on user information.Almost all texts on the Internet are user-generated,and the author's identity of a piece of text is available in most cases.Therefore,this paper proposes the User-enhanced Pretrained Language Models(U-PLMs)for sentiment classification.Specifically,the proposed method injects user ID into the embedding module and the encoder module of a pre-trained language model,respectively.This improves the text modeling and the performance of sentiment classification without modification of the original structure in PLMs.In addition,the proposed framework has great compatibility and can be applied to most existing autoencoder language models.Experimental results show the effectiveness and great performance of U-PLMs.The second work is the study on the quantization of PLMs in task-data-free situations,which is motivated by the excessive size of PLMs and the unavailability of some text data.On one hand,since PLMs are usually too large,these models need to be compressed to be run with reduced cost or on edge devices with limited resources such as mobile phones;on the other hand,due to reasons such as data privacy,it is likely that only a well-trained model is provided for compression while the labeled data it is trained with is not accessible.However,existing works on compression,especially quantization,of pre-trained language models mostly rely on this labeled data,ignoring the privacy of some texts.Therefore,focusing on model quantization,this paper conducts some research on model compression in the situation where task-related data is unavailable and proposes a framework for taskdata-free quantization of PLMs,TDFQ-BERT.The framework uses out-of-domain text which is publicly and easily available as data for training.It also introduces a generator to modify the text with a masking-and-predicting strategy.The generator is trained with adversarial training to generate text which is as valuable as possible for the training of the quantized model under the restriction of fluency.Experimental results show that the proposed framework can quantize the PLMs to a lower precision with only a little drop in performance,even with no task-related data available.In conclusion,on the task of sentiment classification based on user-generated texts,this paper conducts optimization of PLMs based on user information and quantization of PLMs in task-data-free situations.Experimental results on multiple sentiment classification datasets prove the effectiveness of the proposed methods.