Optimization Research On Artificial Neural Tree Network Model With Applications

An artificial neural network (ANN) is a computational model that is inspired by the structure and functional aspects of biological neural networks. A neural network consists of an interconnected group of artificial neurons, and aims to realize some special function by simulating some mechanism of the brain. In most cases, an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network during the learning phase. ANN with its massively parallel processing, fault tolerance, self-organization and adaptive ability, has become a powerful tool to solve complex problems. At present, there exist many NN models, however multilayer feedforward neural network (MFNN) is one of the most popular models which now being studied and applied. It has conspicuous hierarchical structure formed by simple neural units, good nonlinear quality, flexible and effective learning ways and robust simulation capabilities for nonlinear system, so it has been widely used in system identification, data mining, signal processing and fault diagnosis, etc. Although Hornik, etc had proved that MFNN with only a single hidden layer could approximate any complex functions, the method how to find the reasonable structure and the corresponding parameter values for ANN is still a NP-hard problem. Therefore, MFNN still has some problems: first, in designing the topology structure of the neural networks,'experience'or'dynamic'methods which is normally used for determining the number of hidden layers and nodes in each of them according to practical issues has much more uncertainty. Since the generalization ability of the neural networks depends heavily on its structure, traditional design method is prone to cause the generalization ability of neural networks poorly; Second, in optimizing parameters of the neural networks, gradient descent based error-back propagation algorithm supported by powerful mathematics theory still has shortcomings such as slower convergence rate, easier to fall into local optimum and much more sensestive to initial parameter values, etc, greatly limit the application of the MFNN.In consideration of the characteristics of the MFNN, the existing problems and the research trend of related technologies, this paper studies a new tree coding based ANN model, named neural tree network model (NTNM) and its optimization problems, also thoroughly discusses its typical applications in data mining field. The main contents in this paper can be summarized as follows:1. By focusing on the problems encountered in actual application of the NTNM and their characteristics, two aspects in the description for NTNM are studied and improved:(1) In improving the definition of the NTNM, duplicate terminal nodes in the children of a function node which leads to generate more ineffective individuals and maximum depth of the NTNM are solved.(2) Based on the own characteristics of NTNM, a new tuple method for describing the NTNM is presented and it provides a more convenient and scientific description way.2. Through considering the existing optimization method for the topology structure and parameters of NTNM at present, three aspects in its optimization are studied and improved as followed:(1) For optimizing topology structure, building-block-library based genetic programming algorithm, hierarchical variable probability vector based probabilistic incremental program evolution algorithm and tree-encoded based particle swarm optimization algorithm are proposed. All the simulation results show that the above algorithms can effectively reduce the number of invalid individuals generated in evolution process, improve the convergence speed and error precision of the NTNM.(2) For optimizing parameters, differential evolution algorithm is introduced. It has characteristics of less parameters to control, easier to implement and uneasier to fall into local minimum, etc. which make it very suitable for the optimization of parameters.(3) In order to synthesis the optimization of NTNM's topology structure and parameters more reasonable, by considering the learning strategy that optimizes topology structure firstly and then parameters and may cause noisy fitness evaluation problem, an improved breeder genetic programming algorithm is proposed to optimize the topology structure and parameters simultaneously. The simulation results show that this algorithm can effectively improve the convergence speed and error accuracy of NTNM in evolution process.3. By combining ensemble learning and NTNM, the concept of neural tree network ensemble is given and its applications in classification and forecasting belonged to data mining areas are studied:(1) For classification problems, a neural tree network ensemble model which selects NTNM as the basic classifier is proposed. By focusing on the'output combination method', the neural tree network ensemble classification model based on error correcting code with its algorithm design and workflows is given. The validity and superiority of the above model are testing on several UCI data sets. In addition, two neural tree network ensemble classification models based on Bagging and Boosting with their algorithm design and comparison in simulations on several UCI data sets are also showed.(2) For forecasting problems, two neural tree network ensemble forecasting models based on Bagging and Boosting are proposed. By taking the simulation of nonlinear functions as the main application object, their performances are compared with related models, respectively.4. According to the experimental requirements of the NTNM, by combining object-oriented technique with Matlab R2008a in Visual Studio .Net 2008, the experimental platform for NTNM is constructed in C sharp. The main functions of the platform are as follows: experimental data preprocessing, NTNM construction, optimization algorithms of the NTNM integration and experimental results graphical presentation, etc. Finally, the experimental platform is used to solve two prediction problems about housing price index and railway passenger traffic volume.