Regularizing Deep Neural Networks With An Ensemble-based De-correlation Method

Recently,deep neural networks have achieved remarkable success in many fields of application in academia and industry.However,such high-performance models are usually accompanied by some problems,such as complicated model structure and high-dimensional parameters,which makes the model prone to be over-fitting during the training.Therefore,improving the generalization performance of the deep neural network models has becomes a challenge in various research fields.The existing regularization methods for reducing over-fitting mainly include Dropout,DropConnect,and Batch Normalization,but these methods are implicit regularization methods,and without analyzing its performance from the structure of the hidden layer.Because the hidden layer is the key to extract the high-dimensional abstract features in the neural networks,in this paper,we analyze the hidden layer of neural network by utilizing the perspective of the ensemble learning.In fully connected neural networks,the hidden units of the hidden layer are divided into several non-overlapping groups,and each group is regarded as a base learner.In convolutional neural networks,we regard each feature map in the pooling layer as a base learner.By considering the hidden layer of the neural network as an ensemble of several base learners,and then reducing the covariance between these base learners during the training of the network model,the model will be forced to learn more diverse features.Then it can reduce the redundant information existing in the network structure,and improve the generalization performance of the neural network model.We carried out the experiments on the datasets of MNIST,CIFAR-10 and CIFAR-100,respectively.The experimental results show that compared with the conventional regularization methods,our proposed ensemble-based de-correlation regularization method is able to better reduce the redundant information existing in the deep neural network and alleviate the over-fitting,thus improving the generalization performance of the model.