Compression And Optimization On Deep Neural Networks

As the most attractive algorithm in machine learning,deep learning has achieved a lot of amazing performance in some application fields such as computer vision and natural language processing compared with traditional methods.Deep learning is essentially a multi-layer artificial neural network supported by big data.Deep learning is always trapped by the huge amount of parameters and the computation complexity when faced with resource-limited devices such as mobile phones,tablet computer and embedding devices.On the other hand,deep learning models are always over-parameterized.For a specific task,there are always a lot of redundant parameters in the network.To solve this problem,recently a new research filed known as deep compression were studied widely.The main target of deep compression is to reduce the computation cost of deep learning models without the degradation in model performance.The main works of this thesis are:1.A summary of deep compressionAfter reviewing the main methods,models and techniques in deep learning,we widely studied the existing deep compression algorithms and classified them into three categories,namely “Approximation”,“Quantization” and “Pruning”.The “Approximation” methods re-represent the parameter tensors in low-cost ways by tensor decomposition methods.The “Quantization” methods map parameter values from real numbers to limited candidates,the network will share these candidate set later.The “Pruning” methods detect and delete the redundant layers,neurons or connections in the networks and would change the structure of models directly.2.Neuron contribution score evaluation and bias eliminationWe mainly focus on the neuron level pruning in deep compression.Firstly,we show the redundancy of networks in a visible way with the feature map inversion methods.The we generalize three neuron contribution score evaluation methods from previous works.These works could directly used in traditional layer-wise pruning.However,when we consider neurons in different layers,these methods suffer from a significant systematic bias.We propose a simple method to eliminate the bias,makes it possible for global pruning.3.Gradually global pruning framework and the implementationBased on the adjusted neuron contribution scores,we propose a gradually global pruning framework for deep networks pruning.Compared with layer-wise pruning,the proposed methods avoid the difficulty of determining the number of redundant neurons in each layer and greatly reduce the number fine-tuning rounds required.Given a target performance,our framework could automatically find a near optimal network structure.In the experiment part,we implement the proposed methods with the open-source deep learning framework Keras.Our experiment results show the effectiveness of proposed methods.