| With the rapid development of computing power and the exponential growth of data volume,deep learning has made great progress in the fields of computer vision,natural language processing and robotics.However,state-of-the-art deep learning algorithms often rely on the traditional supervised learning paradigm,where data is always kept available during training.This means that when data arrives sequentially,deep neural networks often face the challenge of catastrophic forgetting,usually manifested in a significant drop in model performance on old tasks.Incremental learning was proposed to solve the catastrophic forgetting problem of deep neural networks.This learning paradigm aims to enable the learning system to continuously adapt and learn new concepts while retaining old concept knowledge.The biggest difference between it and the supervised learning paradigm is that The data available to the model at each stage is only from the current task.Based on the above background,this thesis conducts research on the incremental learning method based on the no-sample regularization strategy.The main contributions are as follows:(1)Existing incremental learning methods often suffer from severe classification bias due to the unavailability of old data or the imbalance of old and new samples,and dynamic expansion models are prone to feature confusion.Aiming at this problem,this thesis proposes a K-means Clustering Prototypes Class Incremental Learning algorithm(K-means Clustering Prototypes Class Incremental Learning,KCP-CIL).Specifically,the old class prototypes are sampled by the K-means clustering algorithm in the incremental stage,and the deep features of the old class prototypes and new data are jointly trained to alleviate the catastrophic forgetting of the model.At the same time,it is proposed to reweight the loss function according to the cosine similarity between the two to enhance the distinction between the old class and the new class,and improve the plasticity and stability of the model.(2)The existing mainstream methods of incremental learning mainly focus on largescale networks such as Res Net,Transformer and other architectures,while ignoring the application of incremental learning on mobile terminals and embedded devices.In this thesis,two strategies for transferring knowledge from a large teacher network to a lightweight network in a class-incremental learning scenario are proposed to tackle the problem,given the relatively small amount of literature on lightweight networks.Specifically,static distillation only trains the teacher network on the first task and then enhances the base model of the lightweight network through knowledge distillation.Whereas dynamic distillation guides the lightweight network throughout the incremental learning process,where the teacher network needs to be trained on each task.In this thesis,the KCP-CIL algorithm,static distillation algorithm,dynamic distillation algorithm and the current mainstream no-sample regularization algorithm are compared and analyzed experimentally on the CIFAR100 and Tiny-Image Net datasets.Experimental results show that the algorithm proposed in this thesis achieves the current highest performance on both datasets. |
