Research On Initialization Method Of Convolutional Neural Networks

Convolutional neural network is one of the most commonly used deep learning models.It is inspired by the mechanism of biological visual neuron,and it effectively improves the performance of feature extraction comparing with traditional pattern recognition algorithms.In recent years,models which base on convolutional neural network has been widely used in the field of image classification,object detection,face recognition and natural language processing,and has achieved remarkable results.When training the convolution neural network,the first step is initializing the network.And whether the initialization is appropriate or not significantly affects the convergence speed and performance of training.Therefore,it has important significance in studying on method of initializing convolutional neural network.This paper summarizes the basic concepts and algorithms of the convolution neural network,then expatiates the advantages and disadvantages of the commonly used initialization algorithm such as random initialization and Xavier initialization algorithm.Finally,this paper proposes a method which is named PCA shuffling initialization,and this method can initialize the network effectively by following four steps:(1)First,feed all training samples into the convolutional neural network for forward propagation,and all receptive field in each input feature maps of first convolutional layer are sampled.(2)Analyze the principal component of the sampled image patch in first step,and initialize all convolutional filter using PCA projection matrix(3)Then,shuffling the filter initialized in last step: matrices in all convolutional kernel which convolve with same input feature map are randomly shuffled.(4)Finally,initialize all subsequent convolutional layer by above three steps.At last,in order to verify the performance of the initializing method this paper proposed,various experiments are made on MNIST,CIFAR-10 and STL-10 datasets.The results show that comparing to random initialization and Xavier initialization the method this paper proposed has advantage in accuracy and speed of convergence speed during training the convolutional neural network.