Research On Sentence Representation Learning With Ternary Equilibrium

In the era of the Internet,a large amount of text data is constantly generated in our daily life.These texts contain rich semantic information.How to transform the text information into a form that can be understood by computers has become a hot topic in the field of natural language processing.Sentence Representation Learning studies how to map sentences into the representation vector space with rich semantic information to show the meaning of sentences,so that the language model can be further applied to various downstream tasks of natural language processing.It is difficult to obtain the labeled training data,therefore,training the language model with an unsupervised sentence representation learning method has been a research hotspot.In the current research,most of the studies use unsupervised methods with consistency objective to fine-tune the pre-trained language model,and mainly focus on higher-quality data enhancement methods.Beyond the consistency objective,in this paper we attempt to learn stronger sentence embeddings by further leveraging a novel equilibrium constraint within sentence triplets,and then propose an unsupervised sentence representation learning method based on language model,namely Deep Sentence Representation Transfer with Ternary Angle Equilibrium(DSRTTAE).Specifically,for each sentence without label in the dataset,we select another different sentence to pair it,and then combine the two sentences into a new mixed sentence,so as to generate a sentence triplet with two original sentences and one new sentence.Since the mixed sentence contains information from two original sentences,we suppose that the sentence embedding of the new sentence tends to lie between the two original sentences on the semantic space.With this insight,we regularize the representation of sentence triples in the process of training the language model.Specifically,we instantiate the above ternary relationship from the view of angle measure and suggest a novel regular term,namely Ternary Angle Equilibrium(TAE)term.We also explore a variety of strategies for generating triples and examine the effectiveness of these strategies.At the same time,considering that contrastive learning fits the sentence representation learning task a lot,which can directly and effectively pull sentences with similar meanings closer and push away sentences with large differences,we build a consistency objective to improve the quality of sentence representation.Finally,our proposed DSRTTAE combines unsupervised contrastive learning and TAE regular term to fine-tune the pre-trained language model,and effectively utilizes the semantic features of the data to train a sentence representation model with higher performance.In order to evaluate the performance of our proposed Deep Sentence Representation Transfer with Ternary Angle Equilibrium,we conduct the experiments of Semantic Textual Similarity tasks on STS collection of Sent Eval including STS12-16、STSB and SICK.Compared with the baselines,our methods can achieve better performance.In order to verify the effectiveness of TAE,we also conduct a set of ablative experiments.Results of ablative experiments show that TAE can significantly improve the performance of sentence representation model.The experimental results intuitively show that DSRTTAE,method with considering of ternary angle equilibrium,has stable and reliable performance compared with existing methods.