| The human brain has the ability to remember old information and keep learning new information in new situations.Similarly,continuous learning based on deep neural networks requires learning new knowledge from new samples while retaining old knowledge,which is crucial to the realization of general artificial intelligence.However,when deep neural network is used for continuous learning,the phenomenon of "catastrophic forgetting" often occurs,that is,when learning a new task,the original network parameters will be changed,and the memory of the old task will decline sharply.In addition,continuous learning models usually have a large number of redundant weight parameters,leading to a high demand on computing and storage capacity,which makes it difficult to deploy on resource-constrained devices.In order to solve these problems,the problems of catastrophic forgetting and network pruning optimization in continuous learning are systematically studied.The main work contents are as follows:(1)Aiming at the problem of catastrophic forgetting and the gradual accumulation of new task data in continuous learning,a sample replay optimization method based on the similarity between old and new tasks was proposed.Considering that neural networks learning new tasks have different degrees of review of old tasks,according to the differences in similarity between the old and new tasks,different proportions of typical sample data of old tasks are set to replay--the higher the similarity between the old and new tasks,the fewer replay samples,so as to avoid repeated training,reduce resource consumption and improve learning efficiency.(2)In order to alleviate the difficulty of regularization constraint coefficient setting in catastrophic forgetting,a new regularization coefficient setting method was proposed based on the similarity between old and new tasks.Considering that the neural network learning the new task updates the old task parameters to different degrees,the search range of the regularization coefficient is narrowed according to the similarity difference between the old task and the new task--the lower the similarity between the old task and the new task,the higher the regularization coefficient is,which provides an objective basis for the regularization coefficient setting and realizes the accurate and fast setting of the coefficient.(3)Aiming at the problems of many parameters and large memory consumption in continuous learning model,an adaptive weight pruning method based on gradient was designed.The gradient size was used as the weight importance measurement criterion,and the accuracy decline threshold was used as the guidance basis for adaptive pruning.The structure and parameters of the continuous learning deep neural network were reasonably compressed by the unstructured pruning method.Thus optimize the model performance,reduce the storage occupation and reduce the computation.Through the research of the continuous learning model based on the similarity between tasks and its gradient pruning optimization method,the theory technology and design method related to continuous learning are further extended and improved,and a new solution is provided for alleviating the catastrophic forgetting in continuous learning,optimizing the model structure and improving the network performance. |
