Adaptive Activation Functions In Deep Convolutional Networks

In recent years,deep convolutional networks have made significant breakthroughs in computer vision and pattern recognition.In addition to the advantages of deep structure and convolutional operations,some of the reason come from the development of activation functions.In this paper,the activation of deep convolutional network related to the technology to do more in-depth research.The commonly used Sigmoid activation function and Tanh activation function in the traditional neural network model are prone to gradient disappearance,resulting in the inability of the model structure to deepen.Nowadays,commonly used activation functions are ReLU and its variants,named the rectified unit family.Among them,ReLU(Rectified Linear Unit)activation function easily leads to "neuronal death" Phenomenon;the activation function for solving the "neuronal death" such as PReLU(Parameterized Rectified Linear Unit,PELU(Parameteric Exponential Linear Unit)can learn to adjust its parameters according to different input data during training but will not change in the test phase.Therefore,this paper proposes that the adaptive activation function focuses on solving the problem that the activation function can not respond to different inputs during the test phase.Three forms are mainly designed to combine the basic activation functions to obtain a new activation function.First,the mixed form of activation is the weighted summation of two or more activation functions.The weight is a learned coefficient that remains constant during the test phase.Second,the gated activation form also weighted summation of two or more activation functions,but its weight is the input mapping function,so in the test phase can also adjust the weights according to different inputs,to achieve adaptive activation purpose.Finally,the Hierarchical activation form is a three-layer structure popularized by the gated activation form.After combining multiple basic activation functions,the maximum output value is selected as the activation according to the winner-take-all principle.Finally,this paper verifies the effect of adaptive activation from two aspects: object classification and target detection.Experiments show that the adaptive activation function improves the ability of neural network models to learn nonlinear changes compared with ReLU and other common activation functions and improves the ability of network expression.