Research On Hyperparameter Learning Algorithm Of Stochastic Resonance Threshold Neural Networ

In order to satisfy the constraints of storage conditions and computational complexity,the training of neural networks composed of piecewise linear activation functions or hard limiting activation functions becomes a difficult problem.This is because that these activation functions have no gradient or zero gradients,so the backward propagation training algorithm based on gradient descent cannot be implemented.To solve the above problems,this thesis studies the threshold neural network hyperparameter online learning algorithm based on the mechanism of stochastic resonance.In order to train the neural network composed of threshold activation functions,we first establish a novel activation function model based on stochastic resonance by injecting noise into threshold neurons,which transforms the threshold neuron into a continuously differentiable function.However,the introduction of injected noise extends the noise level into the parameter space of the designed threshold network,leading to a highly non-convex optimization landscape of the loss function.Thus,the hyperparameter on-line learning procedure with respective to network weights and noise levels becomes of challenge.Based on the principle of the adaptive moment estimation(Adam)optimizer,we also carry out moment estimation for the parameters of the injected noises.A stochastic gradient descent algorithm based on stochastic resonance is designed for the designed threshold neural network over parameter online learning.While effectively training the threshold neural network,it also shows the superior learning ability of the proposed hyperparameter online learning algorithm.This thesis also analyzes the efficiency of activation function in the stochastic resonance based threshold neural network.In order to solve the problems of disappearing gradient in saturated region of activation function,insufficient parameter training and no adaptive training of hyperparameter,Parametric Gaussian error linear unit(PGELU)is proposed.PGELU is a novel adaptive activation function with definite physical interpretable property.The high efficiency of PGELU is confirmed by testing PGELU and common activation functions on common data sets.Finally,the quantization of threshold neural networks based on stochastic resonance is also studied.The binarization of activation functions in fully connected networks and convolutional neural networks is experimentally analyzed,and the same fitting or classification accuracy is obtained as that of full-precision networks.This also shows the feasibility of threshold neural networks in edge computing equipment.