Research On The Generalization Performance Of Threshold Neural Network Based On Noise Liftin

The artificial threshold neural network models based on hard limiting activation functions have many advantages,e.g.easy-implementation,low memory storage,low computation costs,etc.However,due to the zero gradient and the non-differentiability of hard limiting activation functions,threshold neural networks are difficult to be trained in practice.The injection of noise makes threshold neural network training possible,and the study of the generalization ability of such neural networks is of more practical significance.This thesis first injects noise into the suprathreshold stochastic resonance model,and derives a kind of differentiable threshold activation function.Three advantages of threshold neural networks are concluded: One is that by injecting noise into the hidden layer neurons,a smaller empirical error loss is obtained than injecting noise into the input data;The second advantage is that it makes a series of non-differentiable activation functions derivable,and the gradient calculation in the backpropagation training of the constructed neural network becomes also feasible;The third is that,in the neural network training,the intensity of the injected noise is adaptively optimized in the same way as the network weight coefficient to obtain the local optimal noise variance and the corresponding probabilistic activation function.Furthermore,under the assumption that the injected noise intensity is small,this thesis expands the error loss function of the noise-boosted threshold neural network in a series of small parameters,and theoretically proves that the noise injected into the hidden layer activation function is equivalent to apply Tikhonov regularization to the error loss function,which effectively improves the generalization performance of the threshold network.Next,through function fitting and regression data set experiments,the adaptive convergence curve of injected noise variance is explored in detail in this thesis.Compared with the classic Sigmoid network,it can be found that the noise-boosted threshold neural networks produce smaller error during both the testing and the training procedure.At the same time,this paper also carries out classification experiments on benchmark image datasets,and the results also proved that injecting noise can improve the data recognition accuracy and generalization performance of the threshold neural network.Finally,we notice that the convergence value of injected noise variance is often greater than unity in functional regression and data classification experiments of threshold neural networks.Through the analysis of the path norm of the probabilistic activation function,the relationship between the upper bound of the Rademacher complexity of the neural network and the noise intensity is given,and a more in-depth theoretical analysis is made on the generalization of the noise-boosted threshold neural network.