Research On Text Similarity Based On Bert

The study of semantic textual similarity is an important task of natural language processing.By accurately recognizing semantic textual similarity,we can achieve more accurate coding and representation of texts such as short sentences and sentences,and further promote the development of other natural language processing tasks.At present,deep network models are the mainstream for processing natural language processing tasks.In terms of semantic textual similarity,the work of deep learning can be basically divided into two categories: one is to modify the network model structure so that the model can better fit the semantic textual similarity problems;the other is learning a universal or unique sentence coding representation,and further predicting semantic similarity through cosine similarity or fully connected layers.The first type of method often has a good effect on a certain type of data set,but the effect has a bottleneck and cannot be generalized to other learning tasks.Another type of method is very convenient for predicting different learning tasks,and the effect is generally good,but it needs to be trained on a large-scale corpus.At the same time,a common sentence representation may ignore the small changes in the sentence order or vocabulary of a sentence,making the true meaning of expression is not well understood.In view of the shortcomings of the current research work,this paper proposes a new model called TSBert based on Bert pre-trained language model that is more suitable for processing semantic textual similarity tasks.The model uses a layered structure: input layer,adaptation layer,coding layer,fusion layer and output layer.The input layer uses the same input processing as the pre-trained Bert model.The adaptation layer uses an adapter to reduce the training parameters of the model,and at the same time is beneficial to update the parameters of the new network layer.On the one hand,the coding layer extracts the CLS vectors existing in the Bert model of the adaptation layer,on the other hand,it adds a new feature processing network,which combines the feature extraction capabilities of RNN and CNN and trains new feature vectors.The fusion layer fuses the two feature vectors of the coding layer with respect to the sentence itself,and is expressed as the feature vector of the semantic textual similarity task.The output layer uses three fully connected layers and the corresponding activation function to make the final feature selection of the feature vector of the fusion layer,and iterates the model parameters through the mean square error function as the loss function.The TSBert model not only considers a general sentence representation,but also constructs a local feature extraction network R-CNN.The combination of the two makes the model perform well on semantic textual similarity tasks.Among them,R-CNN network combines the characteristics of convolutional neural network and recurrent neural network to further identify possible changes in textual semantic.In order to verify the performance of the TSBert model,semantic textual similarity experiments were conducted on the SICK dataset and the STS benchmark dataset.On the set,the Pearson correlation coefficient and Spearman correlation coefficient used for performance evaluation reach the most advanced accuracy in the results of all current papers.