| In recent years,deep learning has undoubtedly been one of the most talked about technologies in artificial intelligence.The powerful automatic representation capability of deep learning has led to substantial breakthroughs in computer vision,social networks,natural language processing,and other fields.Data,models,and computing power are the three core factors that facilitate the development of deep learning technology.The exponential growth of datasets and models has dwarfed the development of hardware computing power.To compensate for the shortage of computing power,stacking multiple hardware accelerators for parallel computing is necessary to support the training of large-scale deep learning models.However,existing technologies have some problems,such as unitary parallelism,underutilization of computational resources,high storage requirements,and inflexibility.In this thesis,we explore distributed model partitioning,communication compression,and memory optimization techniques with large-scale deep learning model training as the research object,and the main research content is as follows:(1)A hybrid parallel training strategy for large-scale deep learning models.Distributed training is one of the most technologies to support large-scale deep learning models.Existing data parallelism and model parallelism methods encounter communication or memory bottlenecks and hence cannot meet the training requirements.Inter-layer pipeline parallelism has low computational resource utilization due to load imbalance.To this end,we analyze the computational and communication overheads under different parallel modes during model training,build a cost model,and search for efficient hybrid parallel partitions and corresponding computational device allocation schemes based on dynamic planning algorithms.The experimental results validate the effectiveness and flexibility of the proposed algorithm by comparing it with current state-of-the-art intra-layer and inter-layer parallelism methods on several benchmark deep learning models and different distributed environments.(2)An intra-layer model parallelism optimization strategy for large-scale Transformer models.The massive parameters and unitary structure characteristics of Transformer models make intra-layer model parallelism the preferred parallel mode for their distributed training.Current intra-layer model parallelism techniques face significant communication overhead.On the one hand,they cannot fully exploit the fine-grained parallelism between computation and communication operations.On the other hand,they do not take advantage of the distribution characteristics of communication data.To this end,we first introduce a fine-grained pipeline execution strategy and design a dynamic programming algorithm to search for optimal operation partitions to maximize the computational and communication overlap.Subsequently,we explore a hybrid communication compression mechanism,which combines Token-level Top-k sparsification and piecewise quantization methods to minimize the amount of communication data.Compared with the benchmark intra-layer model parallelism approach under several different Transformer model configurations,the proposed algorithm significantly improves the training performance while maintaining comparable model accuracy and convergence.(3)A heterogeneous training strategy for large-scale deep learning models.GPU memory capacity is one of the factors limiting the sustainable scaling of large-scale deep learning models.Currently,the dominating memory consumption during the training of large-scale deep learning models shifts from the activation tensor to the parameter tensor,and existing GPU memory offloading methods for model parameters still maintain high GPU memory requirements or use overly aggressive offloading strategies resulting in significant data transfer overhead.For this reason,we first introduced the parameter bucketing mechanism to maximize PCIe bandwidth utilization.Second,considering the asynchronous execution of GPU computation and memory copy,we designed greedy algorithms to determine the offload parameter objects and the offload and prefetch timing,and the model computation graph is reconstructed to make the offload policy effective at runtime.Based on the determined offloading strategy,a parameter preallocation algorithm is proposed to make the parameters that do not coexist during the training process reuse the same GPU memory space,which solves the memory fragmentation problem.Finally,the flexibility and effectiveness of the proposed algorithm are verified on a variety of Bert and GPT models of different sizes.The above research results effectively reduce the demand for computation,communication,and storage resources for efficient training of large-scale deep learning models,alleviate the objective contradiction of unbalanced development speed between model scale and computing power,and help explore the structure and application expansion of largerscale deep learning models. |
