| Although the deep structure guarantees the strong expressiveness of deep neural networks,they may also cause overfitting problems.In order to improve the generalization ability of deep neural networks while retaining their expressivity,researchers have developed many strategies to improve the diversity among hidden units.Following this research direction,we propose a label-based diversity measure(LDiversity)quantified as the gap between a newly added inductive-bias term and a canonical unsupervised diversity measure term by formalizing the effect of the entanglement of the hidden units on the generalization capacity as mutual information.It is also proved that there is an inverse relationship between LDiversity and generalization ability,that is,the reduction of LDiversity usually improves generalization ability.On this basis,a new regularization method called Label-based Diveristy Regularization Method is proposed by using LDiversity regularizers.Its goal is to minimize the classification loss and newly added LDiversity.Based on the general framework of neural networks,this paper conducts comparison experiments with other widely used regularization methods on MNIST,CIFAR-10 and CIFAR-100 data sets.Experimental results show that this method can effectively reduce overfitting and generalization error.This paper also further discusses the role of label diversity measure in network training from the perspective of ensemble learning.It is found that if the hidden unit is regarded as a base learner,the label diversity measure can be used for ensemble learning.As a part of the upper bound of classification error,this measure means that reducing the label diversity measure in ensemble learning can suppress the probability of classification error,and thus an Ensemble-based Label Diversity Method(ELDM)is proposed.Comparison experiments on network frameworks and data sets with different depths show that the ensemble-based label diversity method can effectively reduce overfitting and improve network generalization compared with the method without using it. |
