Improved Distributed Gradient Descent Optimization Algorithms

In recent years,artificial intelligence technology has developed very rapidly,and data storage and mining have achieved great and breakthrough progress.Distributed systems based on big data-cloud computing are a new trend,among which distributed optimization algorithms play a very important role in the development of many distributed systems.In distributed optimization,individuals or nodes communicate through the underlying network,and cooperate to complete the optimization task.This dissertation proposes two improved distributed gradient descent optimization algorithms.The first optimization algorithm mainly considers distributed online optimization problems with constraints.In distributed environment,real-time data flow is optimized by mutual information communication between nodes,so the cost function of each node is time-varying.In order to solve constrained optimization problems effectively,which usually involves expensive projection operation,this dissertation proposes a new distributed adaptive online optimization algorithm based on Frank-Wolfe（DFWAdam）.Adaptive optimization algorithm algorithm（Adam）is an extension of the stochastic gradient descent algorithm.Firstly,in order to effectively overcome the problem of complex calculation of projection operation caused by constraints,the proposed algorithm uses conditional gradient method to replace the expensive projection operation.Secondly,local estimation is updated by local information interaction between each network node.Finally,the convergence analysis shows that the algorithm hasO(T^3/4)Regret bound when the cost function is convex.Numerical experiments using two different types of data sets demonstrate the fast rate of convergence of DFWAdam algorithm.Based on the stochastic gradient descent algorithm,the second optimization algorithm proposed a fast distributed stochastic Nesterov gradient descent algorithm（SFDGND）,which can effectively solve the image classification problem of distributed neural network.The SFDGND algorithm allows data to be randomly and evenly distributed to each node.Each node updates its parameters with a subset of local data,enabling parallel computation.Finally,we compare the effectiveness of the proposed stochastic algorithms in training distributed neural network optimization problems.Experiments on MNIST and CIFAR-10 data sets show that SFDNGD algorithm has better performance in practical applications than existing algorithms.Figure[20]Table[5]Reference[70]...