Improved Sentence Embedding Based On BERT And Prompt-learning

With the rapid development of machine learning,natural language processing has made great achievements in recent years.The research of text semantic relevance has always been the key research object of natural language processing.In the early years,computer scholars expected to model natural language with accurate mathematical models,which catalysed classical models such as word bag,TF-IDF and so on.Later,the rise of deep neural network brought models such as LSTM and GRU,which realized the transformation from traditional method to seq2seq(sequence to sequence).In recent years,the text representation of the pretraining language model represented by Bert has been widely used.The performance of the pretraining language model in multiple natural language tasks has exceeded the methods of word-of-bag,glove,word2 vec and so on.The pretraining language model first carries out the pretraining task on a large scale corpus provide a better initialization on the model weight.On this basis,fine tuning the weight of the model for downstream tasks has become one of the mainstream practices in the field of natural language processing in recent years.However,in the field of text representation,the sentence embedding of non finetuned language model(BERT)has always shown unsatisfactory results.On the one hand,the effect of sentence vector generated by the model itself is not good.On the other hand,although fine tuning the language model can improve the sentence vector representation,the performance requirements and efficiency of standard fine tuning are still not optimistic.Aiming at some shortcomings of the existing work,this paper proposes a text representation scheme based on Bert and prompt learning.The research work of this paper mainly includes the following aspects:1、 Prompt learning input mapping transformation.Transform the Bert input,map the input sentences into the template space,and further extract the sentence vector representation of the pretraining language model through the handcrafted-prompt.2、 Automatically learn template parameters.Switching the discrete template into continuous template allows the model to optimize the template parameters in the training process.Compared with the artificial template,the continuous template has more optimization space inside the template in the training process.The continuous template can also maintain the consistency with the language model in the embedding space,and requires fewer parameters.This practice has certain advantages in the transition from zero-shot to few-shot.3、 The circle loss function is used for specific tasks.For a specific data set,the improved form of circle loss is adopted to strengthen the model’s understanding of the input sentence,so as to improved the corresponding text representation.The improved loss function is helpful to distinguish the positive and negative samples.