Research On Self-Adaption Technology Of The Scale Of Neural Networks

Since the introduction of the neuron model in 1943,neural networks have evolved over the decades.Nowadays,although neural network can handle many complex problems,such as image processing and natural language processing,how to determine the scale of the neural network is still an urgent problem to be solved.The scale of the neural network represents the number of parameters that the neural network contains and the network structure.An oversized neural network not only occupies more storage and computing resources,but also lead to a significant increase in training time and over-fitting problem.A neural network which is too small may not fit the data correctly.A moderately sized neural network can reduce training time and resources as much as possible while guaranteed effect.After deep-learning was put forward,the scale of the neural network becomes more and more difficult to determine due to the deepening of the network.So researchers need a neural network scale adaptive approach to determine the optimal neural network scale.This thesis proposes a method to predict the optimal neural network scale for training a data set by the characteristics of the data set.That means we can use the data dimension of the data set,the dimension of the predicted output of the data set,and the average standard deviation of the data set data to predict which scale is the most suitable for training this data set.In general,complex relationships require a larger neural network,while simple relationships just need a small neural network.The specific approach is as follows:First,collect several data sets,then find out the optimal scale of neural network for each data set by brute force method,then quantify the size of the neural network and the characteristics of the data set,and analyze them by regression.This relationship is ultimately used to predict the optimal neural network scale for the data set.Considering the versatility and representation ability of various neural network structures,this thesis chooses the feedforward neural network to train data sets.Considering the difficulty of nonlinear regression,feedforward neural networks are also used as a tool for regression.In this thesis,the neural network of the training data set is called the basic neural network,and the network used for regression is called the meta network.This thesis collects MNIST,CIFAR and some other data sets to experiment.The result shows that the trained regression model can accurately figure out the optimal scale of the feedforward neural network that a data set needs,verifying that there exist a relationship between data set and scale of network.Using the method of this thesis,before training the feedforward neural network,we can find out the optimal scale of the neural network,reduce the spending of training and avoid the waste of resources caused by excessive scale.