| Distributed deep learning has been widely used to train deep neural networks,especially for large models on massive datasets.Therefore,efficient use of the computing resources and limited bandwidth of nodes in distributed clusters has become the focus of distributed training.This paper studies how to improve the performance of distributed training from two aspects:communication pace and gradient compression.Parameter Server(PS)architecture is the most popular distributed training framework,which can flexibly design the global parameter update manner.However,when scaling to complex heterogeneous clusters,stragglers make it difficult for existing distributed paradigms on PS framework to balance between synchronous waiting and staleness,which slows down the model training.In this paper,we propose Grouping Stale Synchronous Parallel(GSSP)scheme,which groups workers with similar performance together.Group servers coordinate intra-group workers using Stale Synchronous Parallel while they communicate with each other asynchronously to eliminate stragglers and refine the model weights.We further propose Grouping Dynamic Top-K Sparsification(GDTopK),which dynamically adjusts the upload ratio for each group so as to make communication volume differentiated and mitigate inter-group iteration speed gap.We have conducted experiments on LeNet-5,ResNet-18,VGG19 and Seq2Seq with Attention.Results show that GSSP accelerates the training by 46%～120%,with less than 1%accuracy drop.And GDTopK can make up for part of the lost accuracy.In distributed training,the nodes need to transmit gradients frequently to update the global model,which brings huge communication overhead.Gradient compression is currently the mainstream method to solve the problem of excessive communication overhead in distributed training.But the compression rate of most gradient compression algorithms is fixed and not optimized for the actual training process.In this paper,we propose Loss-Voted Compression(LossComp)to select gradients based on realtime changes of Loss during training iterations.The gradient coordinates that help reduce the training Loss in the past several epochs have a greater probability of being selected,and the ratio of gradient sparsification is also changed dynamically.We conducted experiments on multiple models and datasets,and LossComp greatly reduces the communication overhead without affecting the convergence of the model.LossComp even improves the test accuracy of models including LeNet-5,CIFAR10-fast,and VGG. |
