Research On Neural Network Activation Function For Handwritten Character And Image Recognition

The character recognition and image classification are important research directions in artificial intelligence.Through training a set of given input character images and classification labels,the aim can be achieved to predict the classification labels of other input images.Neural network is a kind of autonomous learning image features,and it has an abstract ability at character and voice recognition,image classification,speech recognition,video target tracking and other fields of processing tasks.Although it has an excellent performance,but the larger data amount,the more complex information with the development of"big data"era,which brings challenges to the performance of neural network.In addition,with the improvement of hardware performance,FPGA becomes an effective platform to realize neural network.However,due to the increasingly complex network structure,the hardware resource consumption also increases correspondingly.Aiming at the image classification,handwritten character and Vietnamese character recognition application,this paper makes a deep research on neural network structure and activation function algorithm,which provides reference and technical support for the deep development of neural network in the future.The main research contents of this paper are summarized as follows:On the basis of analyzing the structural characteristics of convolutional neural network(CNN),compared to the unidirectional connection,the CNN?GRU hybrid neural network model is proposed in this paper to issue the interconnection between neurons at the same layer in CNN fully connected layers.In this way,the convolutional layer and pooling layer of the convolutional neural network are used to extract image features.Then,the gated recurrent unit is applied to replace the CNN fully connected layer,which forms a sequential connection relationship between neurons.The dropout technology is employed to optimize the over-fitting phenomenon of the network model.Experiments performed on MNIST handwritten digit dataset show that the recognition accuracy of 99.21%,which is 0.16%higher than Lenet-5 model while the training time and testing time is only 57.91 seconds and3.54 seconds,respectively.In the hardware implementation of neural network,a piecewise linear sigmoid function approximation based on the neurons value's distribution probability is proposed to improve the network recognition accuracy in the low hardware resources condition.When the neural network algorithm is operated in hardware,it usually need to fit the sigmoid activation function to simplify the algorithm complexity.However,the network performance is also decreased with the decrease of the hardware implementation complexity.In order to solve the above problems,a sigmoid piecewise linear fitting method based on the distribution probability of neural network neuron values is proposed.Firstly,the sigmoid function is divided into three fixed regions.Then,the number of sub-segments in these regions is determined according to the distribution probability of the neurons value in these three regions.Finally,the Sigmoid was fitted by using the linear function with slope coefficient of2^-n,and three different fitting behaviors were proposed to be applied to different network layers.Experimental results show that the proposed approximation method achieves high recognition accuracy of 98.42%,68.29%and 63.14%in MNIST,VNCD and CIFAR-10datasets respectively with only an addition circuit,which are 0.57%,2.38%and 1.99%higher than other methods with the same computation complexity.Based on the study of Re LU and its existing modified methods,SWish LU,an improved method of Re LU activation function is proposed to avoid neuronal death caused by Re LU in this paper.This function is activated by Swish function in the negative axis part,so the function has the advantages of both Swish and Re LU.SWish LU function not only can avoid the neuronal death but also reasonably apply the negative input information of the network to improve the network performance.The experimental results show that the network using Swish LU function achieves high accuracy of 99.18%,82.82%and 63.81%in MNIST,VNCD and CIFAR-10 datasets respectively,which are 0.09%,2.08%and 0.74%higher than Re LU function.Aiming at the problem of output deviation is caused by the non-zero output mean of the activation function,an adaptive inverse proportional linear activation(AIPLA)function is proposed.The function uses the piecewise activation method,and each segment has different activation behaviors including linear activation and inverse proportion according to the change of the piecewise point and the slope coefficient.AIPLA has the advantages of the origin symmetry,bounded up and down,non-monotone and simple calculation etc.In addition,the slope coefficient of AIPLA can be learned by self-renewal in the training process,which improve the flexibility of the activation function.Experimental results show that the proposed activation function achieves the highest accuracy of 99.32%,88.79%and65.59%in MNIST,VNCD and CIFAR-10 datasets respectively,which are 0.17%,6.35%and 0.82%higher than the existing activation functions.Meanwhile,under the requirement of the same training accuracy,the training times required by the proposed function are reduced by 4 times,4.5 times and 1.41 times respectively.