Activation Function Awareness Of RNN Algorithm Optimization

Recurrent neural network(RNN)is an important branch of artificial neural network.It achieve efficient processing of sequence data,because RNN is flexible in their use of context information through feedback mechanism of the hidden layer.It has become one of the hot topics in the field of speech recognition,Natural Language Processing,computer vision and so on.On the one hand,recurrent neural network generally uses the S function as the activation function,while the saturation region of the S function limits the convergence speed of RNN training,so the optimization of the activation function becomes a hot research topic.On the other hand,recurrent neural network mainly uses the way of software implementation,the hardware acceleration of the algorithm is of great significance.Aiming at the problems above,the following work was done based on the previous research:1.RNN's theoretical summary research.The Long Short-Term Memory(LSTM)solves vanishing gradient problem through its different gates.The LSTM training includes forward pass and back pass,the activation function and its derivative directly affect the convergence speed of network training.2.RNN's algorithm optimization through optimizing activation function.When the coefficient of S type function is different,the saturation range is different.The extension of unsaturated region is realized by changing parameters.The experimental results show that the optimization method of extended unsaturated region can effectively accelerate the convergence speed.3.RNN's algorithm optimization through the optimization of activation function hardware implementation.It includes error correction term and optimal segmentation based on the property of sigmoid.In accordance with the mapping relationship between different functions,the parameterized sigmoid function and tanh function is implemented,which proved that the hardware implementation of sigmoid has good expansibility.