Research On Cross-Language Sentiment Analysis Method Based On Pre-training Model

The mainstream of sentiment analysis methods based on deep learning usually rely on a large amount of labeled data.However,in practice,due to the limitation of corpus resources,most of the existing sentiment analysis methods are based on English texts,while the research on minority languages which lack corpus resources is less.Moreover,for the same task in different languages,different features engineering is required.As the research on the features of a specific language and the extraction is becoming more and more specified,re-extraction of features takes both time and effort.This research aims to search for a method that can obtain the universal pattern of different languages,and only use tagging data of source language to process sentiment analysis of targeted language with no tagging.In this paper,based on the task of sentiment analysis,cross-lingual learning is carried out through countermeasure training.At the same time,in order to reduce the over fitting and catastrophic forgetting caused by the lack of target language tags,knowledge distillation is used for joint training.Finally,in order to further strengthen the "sentiment" factor of the model,pre training is carried out on a large amount of twitter texts with emoticons,which enables "emoticons" as a bridge to communicate in different languages.The main research work of this paper is as follows.1.Propose a cross-language sentiment analysis method that combines adversarial training and knowledge distillation.Transfer learning is a common method applied to cross-lingual learning in recent years.However,because lacking label data of target domain,when the transfer learning method is applied to BERT(Bidirectional Encoder Representation from Transformers),catastrophic forgetting can be resulted,which will lead to random classification.In order to overcome this problem,we use the Knowledge Distillation method,which is initially used to transfer knowledge from large models to improve the performance of smaller models.We find that this method can be used as regularization to maintain the learning information of the source data,make the generated model domain-adaptive,and avoid the problem of over fitting.2.Propose the E-CLSA model.Generally,the existing machine translation tools to construct pseudo tag data is adopted in the existing cross-lingual research methods,However,this method is unable to capture the specific sentimental knowledge of the target language,which impairs the accuracy of the downstream classification task.Therefore,this paper proposes emoji as a new way to learn sentiment patterns of a specific language,because it is universal in different languages.The multilingual pre training model is used to learn the multilingual knowledge representation in a large amount of unlabeled multilingual data.Through the prediction of emoji,the representation learning,and then the next classification task are carried out.It is proved that this method can greatly improve the performance of the classifier,and has good robustness,even in the case of few labeled data.