Research On Few-shot Learning Based On Pre-trained Token-replaced Detection Models

Few-shot learning is a crucial research area in natural language processing(NLP),aimed at equipping models with human-like generalization ability to learn and adapt to new tasks with only a few examples.Currently,one of the mainstream methods for few-shot learning in NLP involves using pre-trained masked language models,which model downstream tasks as cloze questions by inserting prompts into sentences,thus reducing the objective difference between pre-training and downstream tasks.However,pre-trained masked language models are not necessarily the optimal choice for few-shot learning.In comparison,pre-trained token-replaced detection models have higher computational efficiency and stronger performance,making them a superior few-shot learning solution.Therefore,this thesis proposes a new few-shot learning approach that transforms downstream tasks into token-replaced detection tasks to leverage the potential of pre-trained token-replaced detection models for few-shot learning.Based on this idea,the thesis proposes three specific methods:(1)Few-shot classification method based on pre-trained token-replaced detection models:The main idea of this method is to transform downstream classification tasks as tokenreplaced detection problems.Specifically,first,define a template and multiple label description words for the classification task,and insert them into the input sentence to form a prompt input sentence.Then,use the pre-trained token-replaced detection model to predict the replacement probability of each label description word and select the class corresponding to the label description word with the minimum replacement probability(i.e.,the maximum original probability)as the final prediction result.(2)Few-shot regression method based on pre-trained token-replaced detection models:The main idea of this method is to convert the regression task into a classification task that predicts the probability of a sentence belonging to two extreme categories and then apply the pre-trained token-replaced detection model to the regression task.Specifically,first,design the corresponding template and label description words for the regression task and insert them into the input sentence to form a prompt input sentence.Then,predict the replacement probability of the extreme upper and lower class label description words by the pre-trained token-replaced detection model and calculate the regression prediction value by the integrating formula.(3)Few-shot joint learning method based on pre-trained token-replaced detection models:The main idea of this method is to simultaneously model the fine-grained classification task as a classification and regression problem and conduct joint learning on the above methods.Specifically,design the templates and label description words required for classification and regression of fine-grained classification tasks and add the templates and label description words of classification and regression to the input sentence simultaneously.Then,use the pre-trained token-replaced detection model to predict the classification prediction probability and regression prediction value and fuse them as the final result.Based on this,three different fusion methods are also designed.Extensive experiments show that the three methods proposed in this thesis outperform the few-shot learning methods based on pre-trained masked language models,achieving better performance in single-sentence classification,dual-sentence classification tasks,finegrained classification tasks,and regression tasks.