Fundamental Principles For Fractional Order Gradient Methods

With the development of engineering technology,the idea of " optimization" has penetrated into all kinds of fields.Many science and engineering problems can be trans-formed into optimization problems,such as mathematical modeling,optimal control,neural network training and so on.Gradient method is an important tool in solving all kinds of optimization problems,because of its simple structure,stable performance,and easy implementation.Fractional calculus,a natural extension of integer order calculus,has begun to play an important role in practical engineering applications,especially in the fractional order modeling.Recently,scholars have introduced fractional calculus into the design of gradient methods,and it is found that the fractional gradient method maintains some more superior properties,and has achieved quite a number of success-ful applications.However,the existing research is just in its infancy,and the theoretical basis is not yet perfect.Thus,this dissertation will conduct a comprehensive study of the fractional gradient method from the three aspects of fractional gradient direction,fractional system theory and fractional stochastic perturbation,and initially establish the theoretical framework of the fractional gradient descent,laying a solid foundation for related applications.Firstly,a novel fractional gradient method based on iterative initial value strategy is proposed to guarantee the convergence to the exact minimum point.Furthermore,according to the series representation of fractional differential,the proposed fraction-al gradient method is truncated,resulting in the truncated fractional gradient method which is suitable for general convex functions.The convergence properties are ana-lyzed,and the algorithm is then extended to the(0,2)order and vector cases.Fur-thermore,the concepts of fractional Lipschitz continuous gradient and fractional strong convexity are introduced,and the properties of fractional gradient method for such con-vex functions are studied.Secondly,the system representation of general gradient methods is given,and the fractional gradient method is designed according to some given fractional transfer func-tion,of which the stability analysis is provided.Furthermore,the finite-time gradient method is designed with reference to the idea of finite-time control,which can ensure the finite-time convergence to the minimum point.On this basis,two types of robust finite-time gradient methods are then designed,whose convergence time is robust to initial conditions.Considering that the accelerated gradient method may cause over-shoot and oscillation,the reset gradient method is proposed,which effectively weakens the oscillation phenomenon and significantly accelerates the convergence speed of the original accelerated gradient method.Finally,in order to improve the global convergence ability of the gradient method,the gradient method using Lévy perturbation is proposed.By decomposing the Lévy perturbation into large-step perturbation and small-step perturbation,the Markovian transition property among multiple extreme points is proven.Then,the gradient descent method using truncated Lévy perturbation is proposed,which avoids the difficulty in analyzing small-step perturbation and weakens the conditions for the establishment of Markovian transition property among multiple extreme points.Furthermore,perturbed gradient method with arranged jumping points is proposed,so that the frequency of large-step jump is greatly increased,thereby improving the global search ability.