Research And Application Of The Pretraining Strategies Of Deep Convolutional Neural Netowrk

Deep learning,one of the most hot research fields in machine learning and artificial intelligence,has made ground-breaking improvements on the problems on image classification,video analysis,natural language processing,speech recognition,and even game AI design in recent years.In fact,a deep neural network is able to handle these problems as well as a human being,which makes it widespread concerned by both academic and industrial researchers.However,there still existing two basic problems in the fields of deep learning: 1)the lack of a theory framework of deep learning;2)the need of more effecive deep learning training methods.One of deep learning techniques,pre-training,is highly related to these two basic problems.Pre-training is first proposed by Hinton et.al.in their work of deep brief network,and then been proved effecive by many experiments.A good unsupervised pre-training should consider not only the principle behind the model,but also difficulty of the optimization.The main objective of this paper is to develop a pre-training method for the convolutional neural networks(CNN).All the important progresses in CNN have been summarized in this paper: from network structure design,optimization algorithms to weight initializations.Especially,the motivations,theoratical proofs and algorithm details of all the weight initialization algorithm have been elaborated.In final,an innovative weight initialzation algorithm—PCAUV is proposed.The main contributions of this paper are listed as follows:1.Experimental way to analyze and conclude the variations of convolutional kernels during the training process.2.Experimental way to analyze and evaluate the importance of the convolutional kernels in a well-trained CNN.3.The proposed the PCAUV algorithm for CNN.