Model Optimization And Resource Management For Wireless Distributed Learning Systems

With the development of 5th generation(5G)communication,5G has been applied into various industries,giving birth to many emerging applications related with artificial intelligence(AI),such as intelligent Internet of Things and Internet of Vehicles.To support such emerging applications,wireless distributed machine learning,a promising technology for the next generation communication,has been proposed.Specifically,to protect the data privacy and make full use of the local computing power,mobile devices with heterogeneous data and computing power should first complete the local model calculation.Then,by exchanging the model parameters via the wireless network,the AI model can be trained in a distributed way.However,wireless distributed machine learning still faces a number of key technique challenges,mainly including:(1)The wireless resources and the computing power of devices are limited while the amount of model parameters exchanged during the model training is huge and the computational complexity of the local model calculation is high,which will cause a large learning latency.(2)Since the goals of wireless distributed machine learning and traditional cellular networks are different,traditional wireless resource management algorithms,such as wireless resource allocation,user association,and link selection,are no longer optimal and need to be redesigned.This paper investigates on these above problems,aiming to improve the learning performance from the perspectives of model optimization and wireless resource management.First,considering that a large communication latency will be caused by the large amount of model parameters exchanged during the model training,gradient sparsification is adopted in this thesis,and an adaptive batchsize selection and gradient compression algorithm is proposed.Specifically,the learning performance,i.e.,learning latency and convergence rate,is first analyzed.Then,given the learning latency budget,an optimization problem is formulated to maximize the convergence rate.To solve this problem,the closed-form solutions for batchsize and compression ratio are derived under the given wireless resource allocation.Based on these,a joint batchsize,compression ratio,and wireless resource allocation algorithm is developed.From the analytical results,batchsize and compression ratio are determined by both the local computing power and channel state information(CSI),which is different from the traditional distributed machine learning system.Simulation results show that the proposed algorithm can improve the convergence rate and reduce the learning latency while guaranteeing the learning accuracy.Secondly,to reduce the local computation and communication overhead,model pruning is adopted in this thesis,and an adaptive model pruning and device selection algorithm is proposed.Specifically,the learning performance is first theoretically analyzed,including the learning latency and convergence rate.Then,to improve the learning performance,an optimization problem is formulated to maximize the convergence rate under the given learning latency.Given the device selection,the closed-form solutions are derived for the pruning ratio and wireless resource allocation.Accordingly,a pruning ratio threshold based algorithm is developed for device selection.From the analytical results,different from the traditional cellular network,the wireless distributed machine learning system should select devices according to both the local computing power and CSI.Finally,extensive experiments are carried out to demonstrate that the proposed model pruning algorithm can improve both the convergence rate and learning accuracy.Thirdly,a joint data distribution and CSI based user association strategy is proposed for wireless hierarchical distributed machine learning system.According to the data distribution of devices,two different scenarios are considered,namely,independent identically distributed(IID)and nonindependent identically distributed(Non-IID).For the IID scenario,the analytical results show that user association is only related to the learning latency.By minimizing the learning latency,the optimal user association strategy is proposed,where a mobile device selects the base station with the maximal uplink channel signal-to-noise ratio(SNR).For the Non-IID scenario,user association affects both the learning latency and the model error.The total data distribution distance and learning latency are jointly minimized to achieve the optimal user association and resource allocation.It is shown that both data distribution and uplink channel SNR should be taken into consideration for user association in the non-IID case.Finally,simulation results show that the proposed algorithms can achieve the higher convergence rate and learning accuracy as compared against the traditional schemes.Finally,a learning-oriented device-to-device(D2D)link selection algorithm is proposed for wireless decentralized machine learning system.The effect of link selection on the one-iteration learning cost and convergence rate is first theoretically analyzed,where the learning cost consists of the energy consumption and learning latency.Then,by minimizing the total learning cost,the local computing power allocation,wireless resource allocation,model aggregation weight,and link selection are jointly optimized.Given a link selection,the joint local computing power and wireless resource allocation algorithm and model aggregation weight optimization algorithm are designed,respectively.Based on these,a tabu search(TS)based meta-heuristic algorithm is developed to achieve the feasible solution of link selection.Simulation results show that the proposed algorithm can reduce the learning cost under the given requirement on learning accuracy.These research results in this thesis can provide a theoretical basis for the application of the wireless distributed machine learning system,as well as theoretical support and effective technical solutions for the further evolution of 5G to the 6th generation(6G).