Investigation On Improving Generalization Ability Of Neural Network Based On Information Entropy

Neural network is one of the most important learning model of machine learning. It attempts to learn a mathematical model to describe a sample set, in which the samples are disorderly and unsystematic. Because of BP neural network's structure is simple, algorithm is easy to implement and theoretical solid can achieve a high degree of complex nonlinear mapping, it is widely used in pattern recognition, intelligent control and other fields. However, in the practical application of BP networks, there are also some shortcomings, mainly its slow convergence, prone to over-fitting, thereby affecting the network's generalization ability. Network's generalization ability is the recognition ability for the new sample, is an important indicator reflects the neural network performance.A guiding ideology of improve the network generalization ability is to train a nerual network which in training set can achieve the accuracy requirements and the structure of the network as simple as possible. This paper studies the network structure optimization algorithms, by pruning methods in the process of training network to deleting a number of important units and connections. Emphatically analyzes in the traditional error function to add a penalty term. On this basis, according to slow convergence for the network, prone to over-fitting, designed a penalty term based on information theory. In this paper, we are fused the concept of entropy to the network training process by the regularization method, aimed at improving the generalization ability, while addressing the training efficiency. Finally, some experiments are conducted on synthetic and machine learning data set. The experimental results show that the proposed method can achieve better performance comparing to the standars BP neural network and other other well-known learning methods in the same time complexity.