Research And Application Of Structure Normalization For Deep Network

In recent years,deep learning and neural networks have been widely used in our lives,among which normalization technology has played a pivotal role in promoting the development of deep learning.It normalizes the input in the network to reduce the gradient vanishing and gradient explosion problems in the neural network,but the learning cost of the network is still very high after the normalization operation is added to the neural network,which requires a lot of computer resources to calculate.For this reason,this thesis proposes the structure normalization of deep network,and explores it from two perspectives:network structure and data structure.In terms of network structure.In this thesis,we further propose deep structural weight normalization(DSWN)methods to inject the network structure measurements into the WN to fully acknowledge the data propagation through the neural network.In DSWN,two novel structural measurements are developed to impose regularity on each network weight using different penalty matrices.One is sparsity measurement(DSWN-SM).In this measurement,L_1,2 weight regularization is applied in structural weight normalization model to promote competition for features between network weights,so as to achieve pruning and obtain a sparse network.The other is neuron measurement(DSWN-NM).It uses L₂ norm of column weight to scale up or down the importance of each intermediate neuron,which leads to accelerating the speed of network convergence.Extensive experiments on several benchmark image datasets using fully connected network and convolution neural network are performed,and the proposed DSWNSM and DSWN-NM methods are compared with state-of-the-art sparsity and weight normalization methods.The results show that DSWN-SM can reduce the number of trainable parameters while guaranteeing high accuracy,whereas DSWN-NM can accelerate the convergence while improving the performance of deep networks.In terms of data structure.This thesis proposes a study on decorrelated layer normalization.For Convolution neural network,the proposed method adds Whitening operation to all channels of a single sample,which further reduces the correlation between input features while retaining the original network layer normalization.The data feature expression on the channel has the characteristics of independent and identical distribution,thereby reducing the redundancy of input data and ultimately improving the generalization performance of layer normalization.The experimental results on the CIFAR-10 and CIFAR-100 datasets prove that the method proposed in this paper can improve the accuracy of image classification by 2%to 4%compared with other normalization methods on small batch-size samples.An image recognition system is designed and implemented on the basis of the proposed structural normalization algorithm.Through this system,the superiority of the method proposed in this thesis is shown to readers intuitively.This system is mainly composed of three functional modules:system management module,network training module,and image recognition module.After many image classification results show that the image recognition system constructs a convenient interactive functional interface,realizes the required functions,and verifies the effectiveness of the proposed structure normalization algorithm.