| Stochastic optimization based on stochastic gradient descent(SGD)is widely used and performs well in the optimization of non-convex objective functions such as deep neural networks,and its own randomness helps the model to perform saddle point escape and enhance the generalization performance.As an important hyperparameter,the influence of learning rate on deep learning stochastic optimization is complex.The small learning rate theoretically enables the model to converge smoothly at a small training error level,but it loses the convergence speed,converges slowly,and is easy to fall into local optimization such as saddle points in the later stage.The large learning rate enhances the random perturbation of SGD,increases the convergence speed,and provides the possibility for the model to jump out of saddle points,but the excessive learning rate also brings instability,which is prone to oscillation or even difficult to converge.In order to balance the contradiction between too large and too small learning rate,a variety of learning rate adjustment methods such as learning rate attenuation,cyclic learning rate,and adaptive learning rate have been proposed,which have been proved effective in many practices,but each of them has corresponding shortcomings.The learning rate decay is easy to fall into saddle points when the convergence speed is small in the later stage of training,the loop learning rate will damage the optimization process for a period of time,and the adaptive learning rate is poorly generalized and consumes computing resources.This paper proposes a two-stage learning rate adjustment strategy,which combines the ideas of learning rate decay and cyclic learning rate,aiming to retain their respective advantages while avoiding their respective shortcomings as much as possible,that is,while ensuring the convergence stability of the model,it can also maintain a certain randomness in the late training stage to jump out of local traps such as saddle points,and minimize the damage to the optimization process.The two-stage learning rate adjustment strategy divides the learning rate adjustment into two processes:the training stage and the empowerment stage.The learning rate in the training stage is often attenuated,and the learning rate in the enabling stage increases briefly and is evenly distributed in the middle of the two training stages.Through the theoretical analysis and algorithm setting of each stage,the theoretical effectiveness of the strategy is illustrated.For different algorithms in the enabling stage,a total of eight specific two-stage learning rate adjustment methods are proposed.Through experiments with different models and data,the optimization effects of these eight two-stage learning rate adjustment methods are compared and analyzed with the commonly used learning rate adjustment methods,so as to verify the actual effectiveness of the two-stage learning rate adjustment strategy.Experimental results show that the two-stage learning rate adjustment strategy can improve the overall optimization performance of the model in most cases. |
