Research On User Portrait Algorithm Based On BERT

With the rapid growth of information technology and the further improvement of the national education level,the Internet has become more and more accepted by the public and gradually integrated into people's daily lives.The scale of the Internet has also continued to expand.More and more people are socializing,shopping,and various entertainment activities on the Internet,while leaving a lot of network data.How to divide groups according to the user's personal behavior data left on the network,and accurately and effectively filter out the information required by users from a large amount of information,to meet the individual needs of different users and groups has become a difficult problem to be solved by contemporary enterprises.This is the task of building user portraits,which has attracted more and more attention from enterprises.This article studies an end-to-end deep learning method for building user portraits.The traditional deep learning method is to use the word vector to initialize the first layer of the model,and then complete the user portrait construction by constructing a complex network to extract features.In this way,the generalization ability of the word vector is insufficient,and the second is that the layers behind the word vector are initialized randomly,and training needs to be started from the beginning,requiring a lot of data to make the model converge.In view of the above shortcomings,the research work carried out in this paper is as follows:First,this article completes the task of building user portraits by fine-tuning the Chinese version of BERT,BERT-wwm-ext.BERT used a large amount of unlabeled corpus during pre-training and learned a more general representation,so BERT's word vector generalization ability is very strong.In this paper,the keel architecture based on BERT is modeled,and then modeled by neural network for feature extraction.Since the pre-training weights of BERT are imported,it is eq converge uivalent to that the multilayer neural network has been well initialized,so the model will relatively quickly,and it does not require too much data for training.This article uses the data set provided by the 2016 CCF competition ?Sogou User Portrait Mining in Big Data Precision Marketing?,which includes Sogou users' query terms for one month,and three populations of gender,age,and education label.This paper proposes three models BERTKCNN1,BERTKCNN2,and BERTCATT.Both BERTKCNN1 and BERTKCNN2 use the largest pool of the last four layers of BERT as the embedding layer.Since the user query corpus has no obvious word order,both use CNN to capture the keyword features of the text.Considering that the text length is too long,BERTKCNN1 usesk-max pooling to reduce overfitting.In order to take advantage of more classification features,BERTKCNN2 chooses stitching maximum pooling,k-max average pooling,and pooler_output on the top of BERT as classification features.BERTCATT uses the attention mechanism for the CLS classification labels of each layer of BERT to filter the features that are useful for the task,and uses the multi-sample dropout technology to reduce the risk of overfitting the model.In the experimental part,in order to verify the power of the pre-trained model,this paper sampled more than nine thousand data from the original data set in layers.Under the same conditions,this paper conducts a comparative experiment through BERT and Word2 vec,and the classification accuracy is 16% higher than the latter on average.In addition,under the condition of selecting BERT,the three models proposed in this paper are compared with the other three deep learning neural networks.Experiments prove that the three models proposed in this paper have achieved better results.