Research On Adaptive Activation Function For Deep Learning

Activation function is an important component of artificial neural networks in deep learning.Adaptive activation function control its shape by learnable parameters which is learned jointly with the training of the whole network through back propagation.In this paper,we propose a new method for parameter learning of adaptive activation function.Based on this method,a new activation function Max2 for CNN network and a new activation function MinMax for RNN are designed.Meanwhile we propose a conjecture that each neural network has its own proper activation function,NMPLAAF can be seen as an effective method to find the best activation function of neural networks.It not only greatly reduces the total number of learning parameters of the adaptive activation function in the network,but also speeds up the learning rate of the network,and greatly improves the generalization ability of the network.The new activation function can be applied to existing network structures with very few parameters and computational overhead.The experiment mainly uses MLP,CNN and RNN network structure on image,text and audio data.The experimental results verify our conjecture and show the superiority of our new method.These two new adaptive activation functions avoid gradient dispersion problems,and have higher classification accuracy than other activation functions.