On Improving The Speed Of Deep Learning

Deep neural network has shown excellent improvements in many computer vision tasks such as image classification,object recognition and tracking.However,due to the use of the huge number of parameters,they are unsuitable to be deployed on low-power devices,which is usually memory and computing power limited.As a result,the thesis focuses on improving the speed of deep learning and our main contributions can be summarized as follows:At first,the existing fast deep learning methods are introduced,including parameter pruning and sharing,low-rank factorization,compact filter and knowledge distillation.In view of the method of parameter pruning and sharing,redundancy in the network parameters is seeked and non-critical parameters are removed with the accuracy significantly lowered.The essential parameters of a deep CNN are estimated by matrix decomposition based on low-order factorization,which is computationally expensive.Based on the transferred convolution filter method,a special filter is designed to lower the memory and computational power,which can improve the performance of the network.The knowledge distillation method learns a distillation model and trains a more compact model to replace the original one,which is,however,very limited in practical applications.Then,a fast computing algorithm based on activation function is proposed,namely,self-normalizing piecewise linear units?SPeLUs?for fast approximation of SELUs.There are four steps to achieve SPeLUs:piecewise linear units?PeLUs?in replace of ELUs;SPeLUs by multiplying PeLUs with ?;?_{0 1} and ?₀₁ by solving fixed-point-based equations;ensurance of the spectral norm of J?0,1?smaller than one.SPeLUs adopt piecewise linear functions instead of the exponential part.Experiments show that SPeLUs can provide an efficient and fast alternative to SELUs,with almost similar classification performance over MNIST,CIFAR-10 as well as CIFAR-100 datasets.With SPeLUs,we also show that batch normalization can be simply neglected for constructing deep neural nets,which might be welcome to the fast realization of deep neural network.At last,we focus on the quantization of deep neural networks and propose a binary quantization method by improving trained ternary quantization,which uses only a full-precision scaling coefficient for each layer.These positive and negative weights have same absolute weights value that are trainable parameters.Experiments show that the performance of ETTQ is only slightly worse than TTQ,but can converge faster and more stable during the training over CIFAR-10 and CIFAR-100 datasets.In this thesis,experiments are performed on the Tensorflow framework under Linux operating system.The fast deep learning algorithm introduced in this thesis is simple and effective,and can push the progess of deploying deep neural network at mobile end.