Regularizing Deep Neural Networks By Ensemble-Based Low-Level Sample-Variances Method

Deep Neural Networks(DNNs)with a large number of parameters are very powerful machine learning systems.However,overfitting is a serious problem in such networks.Till now,many regularizers such as L1 regularizer,L2 regularizer and Dropout have been proposed to prevent overfitting.Motivated by ensemble learning,we treat each hidden layer in neural networks as an ensemble of some base learners by dividing hidden units into some non-overlapping groups and each group is considered as a base learner.Based on the theoretical analysis of estimated and traditional generalization error of ensemble estimators(bias-variance-covariance decomposition),we find that variance term in estimated generalization error plays an important role in preventing overfitting and propose a novel regularizer — Ensemble-based Low-Level Sample-Variances Method.For fully connected neural networks,hidden units of each hidden layer are divided into some non-overlapping groups and each group is considered as a base learner.For convolutional neural networks,feature maps of each pooling layer are divided into some non-overlapping groups which are considered as an ensemble of base learners.Based on different partition modes of base learners,we can calculate and bound the value of the variance term in estimated generalization error to prevent overfitting problems.Experiments in FASHION-MNIST,CIFAR-10 and CIFAR-100 show that ELSM can effectively reduce overfitting and improve the generalization ability of DNNs.