Conjugate Gradient Learning Methods With L_1/2 Regularization For Neural Networks

Artificial neural network has become a research hotspot in many fields,due to its superior nonlinear mapping capability,excellent learning ability and wide range of applications.In order to improve the network generalization ability and produce more economical sparse network,and to overcome the disadvantages of neural network learning algorithm based on gradient training,such as the slow convergence speed,falling quickly into the local minima,and easily producing oscillation phenomenon,we mainly study batch conjugate gradient learning algorithm in this pa-per.Firstly,a conjugate gradient method based on the modified secant equation is proposed,and the effectiveness of the method in solving the benchmark problems is verified.Secondly,a con-jugate gradient learning method of double adaptive parameters is proposed,and XOR problem is used to test the algorithm by constructing a neural network.Thirdly,the hybrid conjugate gra-dient method with double adaptive parameters is introduced into the training of BP feedforward neural network,and a conjugate gradient learning method with smoothing L1/2regularization terms is proposed.Finally,for zero-order Takagi-Sugeno fuzzy inference system,a constan-t learning rate conjugate gradient learning method with smoothing L1/2regularization terms is proposed.These global convergent results for these methods are presented in this paper,and showed good performance on numerical results.The main work of this paper is as follows:In Chapter 1,the introduction part reviews the relevant background knowledge of neural network,introduces the research purpose and significance of this paper,and proposes the content of this paper.In Chapter 2,the Dai-Yuan(DY)conjugate gradient method has excellent convergent prop-erties but common numerical performance,therefore a double parameters DY-type conjugate gradient algorithm with the modified secant equation is proposed to improve the DY-type con-jugate gradient.Combined with Wolfe line search,the algorithm can always produce a descent search direction.Since the algorithm makes full use of the information of the gradient informa-tion and function values contained in the modified secant equation,the approximate accuracy of the second-order curvature of the objective function is improved.The algorithm with some proper constants on parameters,exhibits excellent numerical performance.Under reasonable assumptions,the global convergence of the algorithm is easy to establish.In Chapter 3,we mainly deal with how to design a scheme to adaptively update the pa-rameters in the double parameters conjugate gradient method.Based on the assumption that the direction of the conjugate gradient is selected as the quasi-Newton direction in the sufficiently small vicinity of the optimal solution,the information of the calculated gradient,search direc-tion and learning rate is fully utilized to update the parameters.The algorithm generates a new search direction with the quasi-Newton property for the descent direction.Using the XOR prob-lem to test the algorithm by constructing the neural network.The results is shown that the double adaptive parameters conjugate gradient algorithm improves significantly the performance than the other three classes conjugate gradient algorithms under DY framework.Compared with the classical gradient algorithm,the new algorithm also shows good performance.In Chapter 4,we mainly study the training of feed-forward neural network with double adap-tive parameters conjugate gradient method and use it for classification tasks.A double adaptive parameters hybrid conjugate gradient with smoothing L1/2 regularization under a hybrid strate-gy on DY framework is proposed to improve the numerical performance furtherly.The strong Wolfe condition is used to calculate the learning rate.The search directions generated by the hy-brid algorithm still possesses quasi-Newton property and sufficient descent property.As shown in the numerical experiments for five benchmark classification problems from UCI repository,compared with the other conjugate gradient training algorithms,the new training algorithm has roughly the same or even better learning capacity,but significantly better generalization capaci-ty,network sparsity and higher robust.Under mild assumptions,a global convergence result of the proposed training method is proved.In Chapter 5,a batch conjugate gradient neuro-fuzzy learning method with smoothing L1/2 regularization is proposed for zero-order Takagi-Sugeno inference system.An appropriate sparse architecture of the fuzzy system is obtained by using smoothing L1/2 regularization.The constant learning rate in conjugate gradient training method is adopted to reduce the computation cost and improve the learning efficiency.Simulation results show that the new algorithm has stronger sparsity-promoting capability and learning efficiency.Moreover,under mild assumption,the global convergence results is obtained.