Research And Application On Nonlinear Activation Function In Convolutional Neural Network

Convolution Neural Networks(CNNs)based on deep learning has been widely applied inmany fields of computer vision,such as image classification,image retrieval,face recognition,and so on.The CNNs are composed of many important modules,and the nonlinear activation function plays a key role in the CNNs.One of the important factors for the success of the CNNs is the development of the nonlinear activation function.At present,the network architecture based on the improved activation function is tested and evaluated on high quality images,while the images in the real scenario belong to the degraded image,and the research of the degraded image classification is still in the initial stage.First,we study the existing activation functions,and propose a novel parametric activation function.In all activation functions,the Rectified Linear Unit(Re LU)is most widely used because it has the properties of high efficiency and fast convergence.However,Re LU completely discarded negative activation values containing useful information,especially the negative activation values of the shallow layer of convolution neural network.Inspired by Parametric Rectified Linear Unit(PRe LU)and Exponential Linear Unit(ELU),a Power Linear Unit(Po LU)is proposed in this paper.Po LU performs symbolic energy normalization on the negative activation part,and the parameters of symbolic energy normalization can be learned.The proposed Po LU has two strategies: channel-shared and channel-wised.Channel-shared strategy introduces fewer parameters,but channel-wised strategy has better performance.At the same time,Po LU keeps the positive part unchanged as Re LU does.Po LU can be implemented efficiently and flexibly applied to various CNN architectures.Secondly,the important application of Po LU based network architecture is not only in the research of the problem of high quality image classification,but also in the study of the classification of degraded images.The classification of degraded images is of great practical significance.Po LU has a successful application in the classification of high quality images.It is robust to the noise in the data,which is beneficial to the classification of degraded images.There is a lot of noise and serious information loss in degraded images.The performance of degraded images is poor by using pre-trained networks.In this paper,Po LU is applied to thestudy of qualitative image classification,and the Re LU in the original network is replaced by Po LU,and the improved network structure is fine-tuned on the degraded image database.In this paper,we evaluated the NIN architecture based on Po LU on two high quality image classification database,CIFAR10 and CIFAR100.The experiment results show that Po LU is better than Re LU and other corresponding activation functions.At the same time,Po LU is applied to the important task of degraded image classification.The experiments are carried on an improved VGG-M network based on Po LU.The results show that the improved network architecture performs better than Re LU on degraded images.