Attention-based BertCNN: A Method For Text Similarity Calculation

The judgment of text similarity is one of the typical tasks of natural language inference,which has a wide range of practical application scenarios.The difficulty of the task is how to make the model understand the semantics of the text,and calculate the similarity score of the text pair accurately.However,the traditional model based on LSTM or CNN structure are difficult to accurately capture the semantics of the text.The model judgment results can be used to represent the correlation between texts,but it is difficult to measure the semantic similarity accurately.In recent years,with the advent of the pre-trained language model BERT,the field of natural language processing has entered a new era.This paper studies and explores the task of text similarity calculation by virtue of BERT's advantages.The main achievements are as follows:(1)Aiming at the problem that the static word vector used by SiameseCNN can not accurately represent the context semantics,BertCNN is proposed,which introduce the prior language knowledge of pre-trained language model by feature-based approach with BERT to obtain accurate word vector integrating contextual semantics.(2)Considering the Siamese network only extracts the independent features of a single sentence and ignores the the features of the sentence-pair in sentence-coding stage,a BertCNN network integrated the advantages of ESIM is proposed,which make the convolution can extract n-gram features of sentences and the word granularity interaction features between sentence-pair by introducing decomposable attention before convolution to compare the sentence-pair's word granularity similarity.(3)Aiming at the problem the average pooling will treat convolution feature map equally,a BertCNN network with the advantages of ABCNN is proposed,which introduces attention after convolution for the similarity comparing of n-gram granularity between sentences to get the weight of weighted average pooling.(4)Furthermore,combining the advantages of the previous several networks,the Attention-Based BertCNN(ABBertCNN)network is proposed.BERT is introduced.At the same time two attention layers are introduced to extract sentence-pair interaction features from multiple granularity and perspectives.The similarity between sentences is given more attention by the model using the improved pooling strategy.(5)The effectiveness of ABBertCNN is verified through the comparative experiment,which is far better than ESIM and is in line with BERT fine-tuning.At the same time,the training of ABBertCNN is faster than BERT fine-tuning.ABBertCNN is More friendly to lowconfiguration hardware and more suitable for short text.And through the ablation analysis experiment,the contribution of each module to performance improvement is explored.BERT improve the most,both attentions improve the performance further,but the second time is more useful.And the two attention layers can be integrated into the same model.