Research And Implementation Of Social Text Sentiment Analysis Model Based On Deep Learning

In recent years,deep learning and transfer learning represented by pretrained models have been widely used in natural language processing.Finetuning and feature-based are two existing strategies for applying pre-trained models to sentiment analysis.The representative BERT of fine-tuning approach fine-tunes parameters of the pre-trained model on the target task.The representative ELMo of feature-based approach separates the costly pre-trained model from the training of the downstream model.It firstly extracts contextualized embeddings from the pre-trained model and then trains a downstream model on the target task,which saves the computing resources required for training.However,the contextualized embeddings lack rich sentiment information.Besides,training the downstream model only on sentiment analysis suffers from overfitting when labeled data are not sufficien.To improve performance,we show how to apply deep learning methods to build social text sentiment analysis models from three aspects,encoding sentiment information into word embeddings,efficiently capturing temporal relationships between words and efficiently training model.To solve the problem above,we propose a sentiment-contextualizedbased model,called SCe SA and a self-attention-based sentiment analysis model,called SSA.Firstly,in order to generate embeddings with both contextualized and sentiment information,we propose SCe SA based on pretrained model and embeddings refinement method.Secondly,we propose SSA,a self-attention network to substitute the downstream model of SCe SA in order to model text sequences better.Moreover,to generalize better,we improve SSA with an auxiliary task that predicting the sentiment polarity of words,causing SSA to learn multi-tasks jointly.Furtherly,multiple supervised joint training strategies are applied to optimize the SSA network and speed up its convergence.The experiments on SST,IMDB and Sem Eval show that SCe SA and SSA are effective.The main innovations and contributions of our work are as follows:1)Implementation of SCe SA model.The proposed SCe SA applies a lexicon-based embedding refinement method to obtain the sentiment embeddings,which are combined with the contextualized embeddings to generate sentiment-contextualized embeddings.Our experiments show that with CNN as downstream model,SCe SA achieves an accuracy of 48.95%on SST-5,with 1.3% and 1.2% improvement compared to ELMo-SA based on contextualized embeddings and Senti-SA based on sentiment embeddings.2)Implementation of SSA model.To capture long-term dependences of sentence in parallel,we rebuild the downstream model based on a selfattention network.With word sentiment prediction as an auxiliary task,a multi-objective SSA is proposed to achieve better generalization.We define a weighted objective function to balance two tasks and design several training strategies to optimize SSA.Experiments show that the accuracy of SSA on SST-2 is 84.1%,outperforming single-target SSA by 0.8%.Unfreeze training is 0.7% higher than Freeze.3)Comparison with other sentiment analysis models.Experiments are carried out on SST-2,Sem Eval and MR datasets to compare SSA with SAN,a self-attention-based sentiment analysis model,and CNN?SA which uses multi-task learning.Experiments show that SSA outperforms SAN and CNN?SA by 2.1% and 0.7% respectively on SST-2.Besides,SSA is comparable to SAN and CNN?SA on Sem Eval and MR.