An Algorithm For Training Back-propagation Neural Networks Based On Data Parallelism

BP (Back Propagation) algorithm, also known as the error-propagation algorithm, is a widely used training method in the application of neural networks for its fine capability of non-linear approximation. However, it is known to have some defects, such as converging slowly and falling in a false local minimum. Although some optimization algorithm such as RPROP help to speed up the learning process, for the neural networks with tremendous size and extremely large training set these algorithms could not satisfy the demand of implementation.The ability of parallel processing is inherent in neural network, so it is feasible to reduce the long training time with the parallel techniques. There are two different parallel implementation schemes for BP networks, the structure parallelism and the data parallelism. In the data parallelism, the training data is distributed to different computing nodes; each node has a local copy of the complete weigh matrices and accumulates weight change values for the given training patterns, and then the weight change values of each node are summed and used to update the global weight matrices. The data parallelism with a large-grain size reduces the communication time. Therefore, it is mostly implemented in the cluster.By connecting the PCs with a TCP/IP Ethernet local area network, we built up a cluster system with MPI (Message Passing Interface). The parallel BP network is implemented in master/slave mode: The training data is distributed to each slave node; the master node gathers the result processed by the slave and updates the neural network. We also proposed an optimized method which helps to choose better weight matrices to speed up the convergence. In the optimization technique, each node started to train the whole sample set with different initial weights, after several iterations, the node with the minimum error is found and its weight matrices are broadcasted to each node to start the parallel training.We chose the hypertension data provided by the Tongji Medical College of Huazhong University of Science and Technology as the training data, and implemented the parallel BP network to evaluate the proposed algorithm. The experiment shows that, the parallel algorithm tremendously reduces the training time compared to the sequential BP algorithm, and the optimized method increases the speedup and parallel efficiency.