Learning Algorithms Of Growing Sparse Neural Networks

Neural networks have being successfully applied across an extraordinary range of problem domains. However, most of the neural networks now studied are fully connected networks which come across several difficulties in practice such as hard to be implemented in hardware, hard to choose a proper scale and so on. Many new network structures and learning methods are studied to solve these problems. Sparse neural networks are one of them. It has advantages of reduced hardware requirements, improved generalization capabilities and reduced training and recall time.But when using sparse neural networks in practice, it is hard to choose a proper connectivity. Based on new discoveries in brain science, two new learning algorithms are developed which change the network's connection structure at the time of learning, thus an accurate connectivity is not needed. First, we developed the connections-adding learning algorithm which starts from a low connectivity and add connections into the network during learning based on previous learning results. Then, neurons-adding learning algorithm is designed based on previous work. In this method, a maximum connectivity is used to limit the connectivity of network. First train the network using connections-adding learning algorithm. If the connectivity exceeds the max connectivity and the network has not achieved the desired precision, new middle neurons are added. At last, we studied the effect of isomorphic neural networks. When initialize the newly added neurons, take isomorphism into consideration, which improved the fitting precision in the simulation.