Research On Lightweight Deep Learning Technology Using Multi-objective Optimization

The lightweight of the deep neural network is a necessary step for its deployment in edge embedded devices,which includes parameters quantization,model pruning,lightweight con-volutional design and knowledge distillation.Most of the existing model compression meth-ods are defined as a single-objective optimization problem under given resource constraints,such as performance or parameter constraints.The application of deep neural network is limited by performance,computing resources,storage resources and other factors,so model lightweighting naturally has the characteristics of multi-objective optimization.Existing methods do not treat computing and storage resources as independent optimization targets,only a compressed model can be generated one time,which cannot meet the different require-ments of different development scenarios.Compared with the single-objective optimization algorithm,the multi-objective optimization algorithm can address multiple objectives and provide several pareto solutions at once.On the basis of model pruning,this thesis formulate model compression as a multi-objective optimization problem.Firstly,model performance,computing and storage resources are defined as different optimization targets,then,artificial bee colony(ABC)algorithm,knowledge distillation,multi-objective optimization and im-mune algorithm are used to solve this problem to get lightweight deep learning algorithms based on multi-objective optimization.1)Multi-Objective Aggregation Function based artificial bee colony Pruner(MOAFPruner)is proposed.After representing the sub-model structure with the nectar code,a sub-structure parameter sharing training algorithm is used to get appropriate performance for the sub-structure; Then,three search targets,classification accuracy,floating point operations and parameter amount are defined,and three aggregation functions,linear weighting,negative logarithm and exponential decay,are given to convert the three targets into one optimiza-tion target; Finally,the ABC algorithm is improved with three strategies: random generation of code modification number,complementary code initialization and optimal nectar source guided search.It is not necessary to manually set the measurement criteria like the impor-tance criteria based pruning algorithm in our method.The performance of our algorithm has been demonstrated on datasets such as Cifar-10,Cifar-100 and SVHN,which proves that the algorithm can achieve higher compression ratio of floating point operations and parameter amount while mataining the loss of classification accuracy at the same level as ABCPruner,Slimmable Neural Network and Network Slimming on Res Net,VGG and Mobile Net V2.2)Multi-objective Knowledge Distillation based Channel Scalable Pruner(KDCSPruner)is proposed.Unlike removing redundant parameters from a large-size model,this algorithm aims at solving the compression problem of small-size models.Firstly,channel scalable strategy and dynamic residual block is used to increase the number of small-size model's pa-rameters; Secondly,the performance of small-size models is enhanced with multi-objective knowledge distillation algorithm; Thirdly,the ABC algorithm is used to search the sub-structure of the small-size model after its performance is improved; Finally,a compressed model with the same computing and storage resources as the original small-size model but with higher performance is obtained.The performance of the algorithm is certificated on datasets such as Cifar-10,Cifar-100 and SVHN,which proves that the algorithm can effec-tively improve the classification accuracy of small-size models,such as VGG8,Res Net20 and Mobilenet V2,while maintaining the floating point operations and parameters amount at the same or lower level as the original small-size model.Compared with ABCPruner,Slmmable Neural Network and Network Slimming,which can only remove redundant pa-rameters,our algorithm is more feasible by sacrificing a little compression rate in exchange for higher model performance.3)Multi-Objective Nondominated neighbor-based selection Immune Pruner(MONIPruner)is proposed.When applying the model pruning algorithm to different deployment scenarios,it is necessary to repeatedly adjust the hyper-parameters of the pruning algorithm to obtain an appropriate compressed model.In order to solve this problem,our algorithm defines model pruning as a multi-objective optimization problem.Firstly,we define the classifi-cation accuracy,the floating point operations and the amount of parameters as independent targets,and the mathematical definition of the multi-objective model pruning problem is pre-sented; Secondly,Nondominated Neighbor Immune Algorithm(NNIA)is modified to solve this problem.The performance of our algorithm is analyzed on datasets such as Cifar-10,Cifar-100 and SVHN,and the visualization of Pareto fronts of different models on different datasets are presented.Compared with ABCPruner,Slimmable Neural Network and Net-work Slimming,our algorithm can not only get models with the same resource compression ratio and classification accuracy,but the Pareto optimal solution set is obtained.This advan-tage makes our algorithm more flexible in the face of different demands,a single search can satisfy the model's deployment in multiple scenarios.