A Novel Compressed Neural Network Model Based On Automatic Hyper-parameters Optimization

Deep neural networks are currently the basis of many artificial intelligence applications and have led to breakthroughs in lots of areas of artificial intelligence such as the speech recognition and the image recognition.However,the number of levels of deep neural networks is very large and can reach around 5 to 1000 layers,resulting in huge memory overhead for its operation and storage.For such a large-scale neural network model,it is difficult to be applied to mobile devices and adapted to the rapid development of mobile devices today.Therefore,how to compress the neural network volume has become an important research direction in the field of artificial intelligence.This thesis presents a HORD-HD compression neural network architecture by studying neural network compression methods and hyper-parameter optimization methods.At first,this thesis introduces the basic constitution and principles of convolutional neural network.In particular,several simple and practical compression methods are studied,including hash quantization,vector quantization,scalar quantization,and SVD decomposition.Through theoretical analysis and experimental comparisons with other methods,hash quantization has been proved to achieve higher accuracy under the same compression ratio.The experimental results show that the hash quantization method can compress the connection matrix between the last hidden layer and the output layer of a five-layer convolutional neural network to 1/8with less than 1% loss of precision,and so much as to be compressed to 1/16,however,with only loss of 8% accuracy.Second,this thesis studies and analyzes the effects of the Bayesian global optimization method and the HORD(Hyper-parameter Optimization via RBF and Dynamic coordinate search)method on the automatic optimization of the hyper-parameters of the neural networks.Experimental results compared with other methods show that the HORD method can achieve the same optimization effect as other methods with less iterations,and each iteration takes less time.And in the final optimized model,the HORD method achieves 0.1% and 2% error rate improvement over the Bayesian global optimization method.At last,this thesis presents a novel HORD-HD model that analyzes the theory of HORD method to optimize the compression process of hash quantification,describes the steps of compressing the neural network,and designs the related experiments.The experimental results show that HORD-HD can compress a four-layer neural network to 1/16 with loss of 3% accuracy.In conclusion,the experimental results show that approached HORD-HD model in this thesis,the combination method of HORD and hash quantization,caneffectively compress the volume of neural network and provide a feasible solution for the application of bulky neural network model to mobile devices.