A Modified Stochastic Quasi-newton Method For Large Scale Optimization

With the rapid development of modern technology,the scale of various applications of machine learning has also increased sharply.Machine learning is currently considered to be one of the most effective methods to obtain effective information from massive highdimensional data.In the meantime,numerical optimization algorithms are one of the core components of machine learning as a result of the importance on the promotion and application of machine learning model.Nevertheless,many traditional optimization algorithms can not meet the actual demand.Consequently,the requirements for large-scale optimization in modern society are constantly improving.SQN algorithm is a stochastic optimization algorithm based on L-BFGS.During the process of forming inverse approximation Hessian,a collection of L iterates is a basis for the vector s.In SQN algorithm,vector s is the difference of disjoint averages between the 2L most recent iterations.The average can be regarded as a convex combination of w with equal coefficient.We denote arithmetic progression or geometric progression as coefficients of convex combination,thus the modified SQN algorithm is obtained.We denote the modified SQN algorithm as cSQN algorithm.As the coefficients of convex combination vary with arithmetic progression,the modified algorithm is recorded as e-cSQN.By the same token,when the coefficients of convex combination change in geometric progression,the modified algorithm is marked as in-cSQN.Then we analyze the convergence properties of cSQN algorithm.Based on the convergence analysis of standard BFGS algorithm,it is proved that cSQN algorithm still has global convergence and the convergence rate is a sublinear convergence rate.Furthermore,we analyze the computational cost during iterations,whereby we consider the cSQN algorithm is viable.Finally,we test our algorithms on a logistic regression problem through synthetic dataset and real dataset.Under the same conditions,it is found that with synchronous increasing of i and the weight of wi,the convergence rate of ecSQN is faster than SQN algorithm,indicating that wi embodies more information when i increases.However,when the weight difference becomes too large,the convergence rate of cSQN decreases.Sometimes with inverse increasing of i and the weight of wi,the convergence rate of cSQN is the greater than SQN as well,making it clear that the selection of initial points is very important.When comparing the best convergence effect of the algorithms through real dataset,in-cSQN algorithm whose convex combination coefficients change in geometric progression has a faster convergence rate than SQN algorithm and e-cSQN algorithm whose convex combination coefficients vary with arithmetic progression has an equal convergence rate to SQN algorithm.Nevertheless,the computational cost of e-cSQN is cheaper than SQN algorithm during an epoch.Hence the convergence effect of e-cSQN algorithm is better than SQN algorithm.