Neural Network Ensemble Based On Information Theoretic Learning

Multi-layer perceptron and radial basis function neural network are two typicalfeedforward neural networks. They have strong pattern recognition ability and flexiblenonlinear modeling capability. However, in practical applications, the above two types ofnetworks still have some shortcomings. For example, the generalization and anti-noiseabilities of the networks are not satisifing. To improve the generalization and anti-noiseabilities of single multi-layer perceptron and single radial basis function neural network, theinformation theoretic learning based neural network ensembles are studied in this dissertation.In this dissertation, two ensemble strategies of combing neural networks are proposed.The main research works are as follows:1. Neural network ensemble based on the quadratic Renyi entropy is proposed. Thepresented method utilizes Renyi entropy to choose the optimal weights for eachcomponent in the ensemble, which may improve the generalization ability of theensemble network. Moreover, the analytic method rather than the Newtonapproach is utilized to solve the optimization problem of the proposed method.Thus, the computational efficiency can be greatly improved.2. Selective ensemble based on the improved negative correlation learning isproposed. In the proposed method, correntropy is utilized to replace mean squareerror (MSE) in the traditional negative correlation learning. An L1-norm basedregularization term of combination weights is added into the objective function.Moreover, the half-quadratic optimization technique and the surrogate functionmethod are used to solve the optimization problem of the proposed ensemblestrategy.Experimental results demonstrate that the proposed two neural network ensembles basedon information theoretic learning can achieve better generalization ability.