Analysis Of Normalization For Deep Neural Networks

In recent years,with the rapid development of deep learning and neural networks,normalization has been regarded as a key ingredient on the design of deep neural networks.Among various normalization techniques,Batch Normalization(BN)normalizes the input by the mean and variance calculated over the examples in the minibatch to mitigate the gradient explosion and disappearance of deep neural network training.Weight Normalization(WN)accelerates the training convergence by decoupling training parameters to improve optimization problems.Layer Normalization(LN)calculates the mean and variance of all units on a single training sample to normalize the input.With these normalization techniques,the development of deep neural networks makes sense.Therefore,this thesis focuses on the normalization technology and our main contributions can be summarized as follows.(1)Typical normalization techniques in multi-layer feed forward network and convolutional neural network(CNN)are investigated for both performance and implementational complexity,including batch normalization,layer normalization,group Normalization(GN)and weight normalization.This thesis investigates the effect of various parameters of batch normalization algorithm on both the training and testing performance,especially the gain and offset.Based on the experimental observations,a modified version of batch normalization algorithm,called CBN(Convenient Batch Normalization),is proposed.The modified algorithm reduces the complexity of the standard batch normalization algorithm while maintaining almost the same classification accuracy.It is verified by experiments that the use of the proposed normalization algorithm on the 101-layer ResNet network can reduce the training time by 15%.(2)For input of small batch size,a novel normalization algorithm for obtaining more accurate parameter estimation by modified moving mean and variance is proposed.The new algorithm is based on Batch Renormalization algorithm,with a clever adjustment of the gradient update and normalization operations.Experiments show that the proposed normalization algorithm has faster convergence and better performance under small batch inputs.(3)For Long Short-Term Memory(LSTM),various normalization algorithms are compared.Based on the analysis of the suitability of various normalization techniques on CNNs or RNNs,we focus on two tasks(self-encoder and natural language analysis)for experimental validations.Extensive experiment results show that the normalization algorithm suitable for CNN is not necessarily applicable to LSTM.Weight Normalization is more suitable for LSTM tasks for both computational cost and time cost.But if there is less requirement for computational cost,the combination of layer normalization and L1 regularization,with the introduction of smoothing factors,may be the best option at present.