Research On Mixed Activation Function Based On Monte Carlo Tree Search

So far,Deep Learning has been successfully applied to emerging fields such as computer vision.In addition to its excellent convolutional structure,the success of Deep Learning also benefits from the development of activation functions.Therefore,finding a more suitable activation function has become an indispensable research topic in Deep Learning.At present,from the perspective of the use of activation functions in Deep Learning,it is mainly based on the artificially designed activation function ReLU and the variant ReLU activation function proposed to solve the problem of neuron death in the network training process of the ReLU function.The use of mixed activation functions generated by search construction technology is relatively rare,and related research is also relatively lacking.For different types of activation functions,their nonlinear expression ability is different,and the improvement of the network learning effect is also different.In view of this,this paper has carried out the research of mixed activation function,the main research work is as follows:(1)By studying the structural characteristics of the activation functions GELU?Swish and Mish,this paper proposes a mixed activation function structure in which the original input is multiplied by the composite function modulation structure,and this paper combines the mathematical expression of the LSTM and GRU gating structure with the mixed activation function structure,the mathematical expression of is mapped to determine the meaning of the mixed activation function structure proposed in this paper.(2)Based on the mixed activation function structure,this paper constructs a variety of mixed activation functions through Monte Carlo Tree Search technology and performs balanced screening of these activation functions under a variety of conditions,and finally selects the activation function Mash,which has a simple structure and can retain negative information,as the mixed activation function proposed in this paper,in addition,this paper analyzes the image characteristics of the Mash function and studies the correlation between this function and other activation functions,verifying that this function can not only effectively avoid problems such as neuron death,but also make the network better expressive ability.(3)In the phase of experimental comparison and verification,this paper applies the Mash function and other activation functions to different types and different depths of neural network models for verification experiments on a variety of data sets.The experimental results show that the mixed activation function Mash proposed in this paper can not only make the network converge better in the training process,but also has universal applicability.(4)In the research phase of the Mash function parameters,this paper expands the parameters of the Mash function,conducts research and experimental analysis on the expanded parameters and summarizes the role of the expanded parameters in the network training process.